Prepare the repository for a public release
- docs/deployment.md: step-by-step Docker deployment for anyone (setup, configuration, LAN/HTTPS/VPN access, adding manga, operation, prebuilt images, backups, peering, troubleshooting, rules for agents). - deploy/circus.sh: host helper without Node (status, up/down with queue pause, logs, pause/resume, update, rescan, restore-test); deploy/compose.https.yml + Caddyfile for automatic HTTPS. - README rewritten for public readers (Docker first, sources/copyright note, docs index); AGENTS.md + CLAUDE.md for contributors and coding agents. - Removed notes about one particular installation (Windows host, handoffs, audits of a private library, benchmarks with local paths, wishlist, reading map) and the unused ops/proxmox templates; generalized remaining mentions of hosts, VPN and Windows-only commands. - Defaults: TZ and reading time zone UTC; .gitattributes keeps shell/YAML LF. - Personal agent coordination files are git-ignored (CLAUDE.local.md, .claude/skills/manga-org/). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
7b52da2f9c
commit
f53b92a7dc
24 files changed
+556
-122
No files matched your search
+7
-16
@@ -13,7 +13,7 @@ Related documents:
|
||||
- [scraper-service.md](scraper-service.md): managed queue, private API, and adapter contract.
|
||||
- [collections.md](collections.md): relations, groups, curator UI and agent research.
|
||||
- [resilience.md](resilience.md): supervision, recovery limits and incidents.
|
||||
- [compose-deployment.md](compose-deployment.md): local/VM deployment and release workflow.
|
||||
- [deployment.md](deployment.md): Docker deployment, HTTPS, operation, updates and backups.
|
||||
- [settings-navigation.md](settings-navigation.md): User/Admin workspace, scraper tabs and deep links.
|
||||
- [peering.md](peering.md): optional private server-to-server catalog and transfer.
|
||||
- [discovery.md](discovery.md): reader-owned, opt-in deterministic discovery feed and its limits.
|
||||
@@ -325,20 +325,11 @@ mode = adapt && intended ∉ compatible ? compatible[0] : intended
|
||||
|
||||
## 8. Deployment
|
||||
|
||||
### Current Windows deployment
|
||||
|
||||
The observed local installation uses the Windows Scheduled Task
|
||||
`\MangaCircus Supervisor`. At logon it runs `tools/run-supervisor.ps1`, which appends
|
||||
logs under `data/scraper/` and starts one `tools/start-all.js` supervisor. That parent
|
||||
owns exactly one reader (`server.js`, port 8080) and one scraper daemon
|
||||
(`scraper/daemon.js`, private port 8091). Starting `npm run start:all` manually at the
|
||||
same time would create conflicting writers and is not supported.
|
||||
|
||||
The scraper's authoritative `data/scraper/queue.json` is schema version 3. It supports
|
||||
legacy archive checkpoints and paused conversion of completed legacy work while adding
|
||||
source-aware lanes, scoped queue jobs, incremental chapter publication, and nightly catalog discovery. See
|
||||
[local-deployment.md](local-deployment.md) for operational checks and restart/recovery,
|
||||
and [scraper-service.md](scraper-service.md) for the queue/API contract.
|
||||
The step-by-step guide is [deployment.md](deployment.md). The scraper's authoritative
|
||||
`data/scraper/queue.json` is schema version 3. It supports legacy archive checkpoints
|
||||
and paused conversion of completed legacy work while adding source-aware lanes, scoped
|
||||
queue jobs, incremental chapter publication, and nightly catalog discovery. See
|
||||
[scraper-service.md](scraper-service.md) for the queue/API contract.
|
||||
|
||||
### Docker or manual deployment
|
||||
|
||||
@@ -352,7 +343,7 @@ docker compose up -d --build
|
||||
- `/data`: reader state must be writable; the scraper also needs its writable `/data/scraper` state.
|
||||
- `/backup`: reader only, writable target of the built-in backup (`BACKUP_PATH`; off when unset).
|
||||
- **Configuration:** process wiring uses environment variables (`PORT`, `LIBRARY`, `DATA`, `BACKUP`, `SIGNUP`, `SCAN_INTERVAL`, `FULL_SCAN_INTERVAL`, `COOKIE_SECURE`, `SCRAPER_URL`, and `SCRAPER_TOKEN`). Queue pacing and automatic-update schedule are persisted in `data/scraper/queue.json` and edited through the authenticated admin UI.
|
||||
- **Without Docker:** `npm run start:all` with Node ≥ 18, or supervise `npm start` and `npm run start:scraper` independently. On the current Windows machine that supervision is already owned by `\MangaCircus Supervisor`.
|
||||
- **Without Docker:** `npm run start:all` with Node ≥ 18 starts one supervisor that owns exactly one reader (`server.js`, port 8080) and one scraper daemon (`scraper/daemon.js`, private port 8091), or supervise `npm start` and `npm run start:scraper` independently. On Windows, `tools/run-supervisor.ps1` runs `start:all` with logs under `data/scraper/`, e.g. from a scheduled task at logon. Never start a second supervisor or a container against the same data.
|
||||
- **Behind a reverse proxy:** terminate TLS, set `COOKIE_SECURE=1`. A sub-path works.
|
||||
- **Upgrades:** replace the code and restart. Reader state has no explicit migration;
|
||||
unknown fields in `db.json` are kept. The scraper normalizes old queue entries into
|
||||
|
||||
+2
-2
@@ -112,5 +112,5 @@ and queue changes made after that backup are lost.
|
||||
|
||||
The scraper token is not in `DATA`; Compose keeps it under `.secrets/` (or the
|
||||
deployment's `SCRAPER_TOKEN_PATH`). Restore tests should unpack into a separate
|
||||
folder and run an isolated stack (no second writer against the live library), as
|
||||
a restore-test script does for the host-level archives.
|
||||
folder and run an isolated reader (no second writer against the live library), as
|
||||
`deploy/circus.sh restore-test` does.
|
||||
+3
-3
@@ -26,7 +26,7 @@ selection and must use **Validate & save JSON**, not an ordinary collection save
|
||||
Metadata imports/research are blocked while drafts exist so they cannot silently
|
||||
replace them. Relationship edits save immediately when there is no collection draft.
|
||||
|
||||
Creation works on HTTPS, localhost and ordinary HTTP LAN/Tailscale addresses. The
|
||||
Creation works on HTTPS, localhost and ordinary HTTP LAN/VPN addresses. The
|
||||
browser uses secure random bytes when its secure-context-only `randomUUID` API is
|
||||
unavailable. This does not change networking or the server's revision/validation rules.
|
||||
|
||||
@@ -115,9 +115,9 @@ Provider contract: [AniList relationship connections](https://docs.anilist.co/gu
|
||||
|
||||
## Repeatable functional acceptance
|
||||
|
||||
Run `npm.cmd test`, build `docker compose build reader`, then:
|
||||
Run `npm test`, build `docker compose build reader`, then:
|
||||
|
||||
```powershell
|
||||
```bash
|
||||
# Playwright is optional developer tooling; not a deployed application dependency.
|
||||
# Set NODE_PATH to its installed package root if it is not on normal resolution paths.
|
||||
node tools/test-compose.js --collections-ui --settings-ui
|
||||
|
||||
@@ -0,0 +1,194 @@
|
||||
# Deploying MangaCircus with Docker
|
||||
|
||||
This guide sets up MangaCircus on one Linux host with Docker Compose: the reader (web app), the managed
|
||||
scraper, and optionally HTTPS and server peering. Every step is a command with what you should see. It is
|
||||
written so that a person or a coding agent can follow it without other context.
|
||||
|
||||
## 1. What runs
|
||||
|
||||
| Service | Port | Role |
|
||||
|---|---|---|
|
||||
| `reader` | 8080 (published on `READER_BIND`) | Web app: library, reader, accounts, settings. The only writer of users, progress and the catalog (`DATA_PATH`). Reads the library read-only |
|
||||
| `scraper` | 8091, internal only | Download queue and source adapters. The only writer of the library (`LIBRARY_PATH`) and of its own state (`SCRAPER_DATA_PATH`). Never publish or proxy this port |
|
||||
| `peer-gateway` | 8093, optional (`--profile peering`) | Shares chosen manga with other MangaCircus servers over mutual TLS ([peering.md](peering.md)) |
|
||||
|
||||
All services run from one image (`node:24-alpine`, no runtime dependencies) as uid/gid 1000. Each container
|
||||
has a read-only root file system, no capabilities, rotated logs, a health check and an in-container
|
||||
supervisor that restarts a crashed process. The reader works when the scraper is stopped.
|
||||
|
||||
Data on the host, all bind mounts (no named volumes):
|
||||
|
||||
| Path (`.env`) | Contents | Back up? |
|
||||
|---|---|---|
|
||||
| `LIBRARY_PATH` | The manga: `Manga/Book/Chapter/images` ([library-format.md](library-format.md)) | Yes, it is your collection |
|
||||
| `DATA_PATH` | `db.json` (users, progress, votes), catalog, library index cache | Yes (built-in backup, section 8) |
|
||||
| `SCRAPER_DATA_PATH` | Queue, settings, source ledger, metadata cache | Included in the built-in backup |
|
||||
| `BACKUP_PATH` | Built-in backup archives | Keep it on another disk or NAS |
|
||||
| `SCRAPER_TOKEN_PATH` | Shared secret between reader and scraper | Regenerate if lost |
|
||||
|
||||
## 2. Requirements
|
||||
|
||||
- Linux x86_64 or arm64 with Docker Engine 24+ and the Compose plugin 2.24.4+ (`docker compose version`).
|
||||
Docker Desktop on Windows or macOS works for trying it out. Bind mounts are slow there for large libraries.
|
||||
- 1–2 CPU cores and 1 GB RAM are enough for a few readers. The first scan of a large library is
|
||||
I/O-bound (about 14 minutes for 300,000 pages on a NAS over NFS). Later starts reuse the index cache.
|
||||
- Disk: the library plus room for downloads in progress. The image is about 60 MB.
|
||||
- Git to fetch the code. Node is **not** needed on the host.
|
||||
|
||||
## 3. Get the code and prepare folders
|
||||
|
||||
```bash
|
||||
git clone <repository-url> mangacircus && cd mangacircus
|
||||
mkdir -p library data/scraper data/peering data/peering-usage backup .secrets
|
||||
openssl rand -hex 32 > .secrets/scraper_token && chmod 600 .secrets/scraper_token
|
||||
cp .env.example .env
|
||||
sudo chown -R 1000:1000 library data backup
|
||||
```
|
||||
|
||||
The containers run as uid 1000: the reader must be able to read the library and write `data/`, and the
|
||||
scraper must be able to write the library. On an NFS share with squashed ids, map all users to one account
|
||||
that owns the files. Don't use `chmod 777`. Bind mounts never create missing folders: a typo in a path
|
||||
fails the start instead of showing an empty library.
|
||||
|
||||
Already have a library? Point `LIBRARY_PATH` at it (folder format in [library-format.md](library-format.md)).
|
||||
Starting empty? Fill it from the web UI's scraper queue (section 6). To try the reader first, create five
|
||||
demo manga in an empty library after the build in section 5:
|
||||
`docker compose run --rm --no-deps scraper node tools/make-dummy.js /library`.
|
||||
|
||||
## 4. Configure `.env`
|
||||
|
||||
| Variable | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `LIBRARY_PATH` | `./library` | Library folder (absolute path recommended) |
|
||||
| `DATA_PATH` | `./data` | Reader data |
|
||||
| `SCRAPER_DATA_PATH` | `./data/scraper` | Scraper state |
|
||||
| `PEER_DATA_PATH`, `PEER_USAGE_PATH` | `./data/peering`, `./data/peering-usage` | Peering identity and transfer counters (folders must exist even when peering is off) |
|
||||
| `SCRAPER_TOKEN_PATH` | `./.secrets/scraper_token` | The secret file from section 3 |
|
||||
| `BACKUP_PATH` | unset = backups off | Folder for built-in backups (section 8) |
|
||||
| `READER_BIND` / `READER_PORT` | `127.0.0.1` / `8080` | Where the reader listens on the host. `0.0.0.0` = every interface (plain HTTP, LAN only) |
|
||||
| `TZ` | `UTC` | Time zone of the containers; affects the nightly update time and backup schedule |
|
||||
| `COOKIE_SECURE` | `0` | `1` when readers reach the site over HTTPS |
|
||||
| `READING_WINDOWS_JSON`, `READING_TIMEZONE`, `READING_QUIZ_*` | off | Optional server-enforced reading hours ([reading-limiter.md](reading-limiter.md)) |
|
||||
| `APP_IMAGE`, `VCS_REF` | `mangacircus:local`, `development` | Image to run; set a pinned registry image to deploy prebuilt releases (section 7) |
|
||||
| `COMPOSE_FILE` | `docker-compose.yml` | Extra Compose files, `:`-separated (e.g. `docker-compose.yml:deploy/compose.https.yml`) |
|
||||
| `MANGACIRCUS_DOMAIN` | – | Host name for `deploy/compose.https.yml` |
|
||||
|
||||
Sign-up is closed: the first account becomes the admin, who then adds the other users. Reader settings
|
||||
not listed here (scan intervals, scan concurrency) are in the README's environment table.
|
||||
|
||||
## 5. Build and start
|
||||
|
||||
```bash
|
||||
docker compose config -q # no output = the configuration is valid
|
||||
docker compose build
|
||||
bash deploy/circus.sh up # reader first, waits for its health check, then the scraper
|
||||
bash deploy/circus.sh status
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
```
|
||||
SERVICE STATUS PORTS
|
||||
reader Up 1 minute (healthy) 127.0.0.1:8080->8080/tcp
|
||||
scraper Up 40 seconds (healthy) 8080/tcp
|
||||
{"ok":true,"scan":{"mangas":…,"stale":false,"error":null}, …}
|
||||
{"status":"idle","paused":false,"active":0,"queue":0, …}
|
||||
```
|
||||
|
||||
Open `http://127.0.0.1:8080` (or an SSH tunnel: `ssh -L 8080:127.0.0.1:8080 <host>`) and create the admin
|
||||
account. The library page fills as soon as the first scan finishes.
|
||||
|
||||
## 6. Access and adding manga
|
||||
|
||||
- **On the LAN:** set `READER_BIND=0.0.0.0` (or the host's LAN address), or use `compose.lan.yml` with
|
||||
`READER_LAN_BIND`. This is plain HTTP: logins and cookies travel unencrypted, so never forward it to the
|
||||
internet.
|
||||
- **HTTPS with the bundled Caddy:** point a DNS name at the host, open ports 80 and 443, set
|
||||
`MANGACIRCUS_DOMAIN=manga.example.org` and `COMPOSE_FILE=docker-compose.yml:deploy/compose.https.yml`,
|
||||
then `bash deploy/circus.sh up`. Caddy fetches the certificate. The reader gets no host port and secure
|
||||
cookies.
|
||||
- **Your own proxy:** keep `READER_BIND=127.0.0.1` and proxy `http://127.0.0.1:8080`. Alternatively, use
|
||||
`compose.vm.yml`, which joins the reader to an external Docker network (`PROXY_NETWORK`, default
|
||||
`mangacircus-edge`) and removes the host port. Set `COOKIE_SECURE=1`. All pages use relative URLs, so a
|
||||
sub-path such as `/manga/` works.
|
||||
- **Private access over a VPN** (WireGuard, Tailscale, …): bind the reader to the VPN address, or keep it on
|
||||
loopback and forward from the VPN client.
|
||||
- **Adding manga:** admins use **Settings → Admin → Scraper → Inspect & queue** with a series URL from a
|
||||
supported source (WeebCentral, MangaDex, MangaPill), or copy folders into the library by hand. The reader
|
||||
notices new folders within `SCAN_INTERVAL` minutes. Rules for tools that write into the library:
|
||||
[scraper-guide.md](scraper-guide.md). Only download what you are allowed to copy; the scraper keeps
|
||||
conservative per-host rate limits and stops on 401/403/429 ([scraper/AGENT_RUNBOOK.md](../scraper/AGENT_RUNBOOK.md)).
|
||||
|
||||
## 7. Operating it
|
||||
|
||||
`deploy/circus.sh` wraps the safe sequences (run it from any folder; it finds the repository itself):
|
||||
|
||||
| Task | Command | Notes |
|
||||
|---|---|---|
|
||||
| Status | `bash deploy/circus.sh status` | services, reader health, queue |
|
||||
| Stop / start | `… down` / `… up` | `down` pauses the queue and waits for the running chapter (up to 15 minutes); `up` resumes only what `down` paused |
|
||||
| Restart | `… restart` or `… restart reader` | |
|
||||
| Logs | `… logs scraper 200` | |
|
||||
| Pause / resume downloads | `… pause` / `… resume` | Same as the buttons in Settings → Admin → Scraper |
|
||||
| Update to the newest code | `… update` | `git pull --ff-only`, rebuild, pause, recreate, resume |
|
||||
| Re-read edited `chapter.json` files | `… rescan` | Full reconciliation without an admin login; admins can also use Settings → Admin → Rescan |
|
||||
| Check the newest backup restores | `… restore-test` | Throwaway reader without network; prints `RESTORE TEST OK` |
|
||||
|
||||
**Deploying prebuilt images** (several hosts, or no building on the server): build once, push to your
|
||||
registry and pin the digest:
|
||||
|
||||
```bash
|
||||
docker build --build-arg VCS_REF=$(git rev-parse --short HEAD) -t registry.example.org/mangacircus:$(git rev-parse --short HEAD) .
|
||||
docker push registry.example.org/mangacircus:<commit> # note the sha256 digest it prints
|
||||
# on the server, in .env: APP_IMAGE=registry.example.org/mangacircus@sha256:<digest>
|
||||
docker compose pull && bash deploy/circus.sh restart
|
||||
```
|
||||
|
||||
Pinned digests make rollbacks exact: put the previous `APP_IMAGE` back and restart. Data formats only grow
|
||||
(new fields are ignored by older releases), but a rollback does not undo queue or catalog changes made by
|
||||
the newer release.
|
||||
|
||||
**Rules for operators and agents:**
|
||||
- Never run two scrapers, or a container and a native `node` process, against the same library or data
|
||||
folder.
|
||||
- Never run `docker compose down -v`. Stop with `deploy/circus.sh down`.
|
||||
- Never publish port 8091, and never print or commit `.secrets/` or `.env`.
|
||||
- One-off maintenance tools that write `chapter.json` files are dry runs by default. Run them with
|
||||
`--apply --writers-stopped` only after `docker compose stop scraper`, then `deploy/circus.sh rescan`.
|
||||
Examples: `tools/backfill-chapter-dates.js` and `tools/release-dates.js`, run inside the image with
|
||||
`docker compose run --rm --no-deps scraper node tools/<tool>.js /library …`.
|
||||
|
||||
## 8. Backups
|
||||
|
||||
- **Reader and scraper data:** set `BACKUP_PATH` to an existing folder on another disk or NAS (owned by uid
|
||||
1000), restart, and choose the schedule and retention under **Settings → Service health → Backups**. A
|
||||
run pauses the queue, waits for the current chapter, archives the data folder without stopping anything
|
||||
and resumes. Details, archive contents and the restore procedure: [backup.md](backup.md). Check it now
|
||||
and then with `deploy/circus.sh restore-test`.
|
||||
- **The library:** back it up with your usual file backup (it is plain folders and images), including
|
||||
hidden `.mcscraper-*` folders if downloads are in progress.
|
||||
- **Restore:** `deploy/circus.sh down`, move the data folder aside, unpack the archive into a fresh data
|
||||
folder (`tar -xzf circus-….tar.gz -C data`, owned by uid 1000), `deploy/circus.sh up`. Restored archives
|
||||
have the queue paused; resume it after checking.
|
||||
|
||||
## 9. Optional: server peering
|
||||
|
||||
Share selected manga with friends' MangaCircus servers: enable the `peering` profile
|
||||
(`docker compose --profile peering up -d peer-gateway`) and follow [peering.md](peering.md). The gateway
|
||||
needs its own reachable port (`PEER_BIND`, `PEER_PORT`) and uses mutual TLS with pinned certificates.
|
||||
|
||||
## 10. Troubleshooting
|
||||
|
||||
| Symptom | Cause / fix |
|
||||
|---|---|
|
||||
| `docker compose up` fails with "bind source path does not exist" | A folder from `.env` is missing; create it (section 3) |
|
||||
| Reader healthy but the library is empty | First scan still running (`docker compose logs reader` shows `[scan] …` when done), or `LIBRARY_PATH` points at the wrong folder |
|
||||
| Scraper jobs fail with `EACCES` | The library or `SCRAPER_DATA_PATH` is not writable by uid 1000 |
|
||||
| Backups fail with `EACCES` on a NAS | Map the share's users to the uid that owns the folder; the backup probes with a real write |
|
||||
| Settings → Scraper says the service is unavailable | Scraper container stopped or unhealthy: `deploy/circus.sh status`, `logs scraper` |
|
||||
| Edited `chapter.json` changes don't show | Run `deploy/circus.sh rescan` (in-place edits are picked up by full reconciliations, every 24 hours by default) |
|
||||
| Logins fail behind HTTPS | `COOKIE_SECURE=1` needs HTTPS end to end at the browser; with plain HTTP keep it `0` |
|
||||
|
||||
More background: [architecture.md](architecture.md) (components, API, security model),
|
||||
[import.md](import.md) (scanning and caching), [scraper-service.md](scraper-service.md) (queue, adapters,
|
||||
recovery) and [resilience.md](resilience.md) (supervisors and incidents).
|
||||
+2
-2
@@ -49,11 +49,11 @@ No scraper queue edit occurs.
|
||||
|
||||
From the project directory:
|
||||
|
||||
```powershell
|
||||
```bash
|
||||
node discovery/cli.js run --dry-run --data data --show
|
||||
npm run test:reader
|
||||
docker build -t mangacircus:discovery-test .
|
||||
$env:APP_IMAGE='mangacircus:discovery-test'; npm run test:discovery-ui
|
||||
APP_IMAGE=mangacircus:discovery-test npm run test:discovery-ui
|
||||
```
|
||||
|
||||
The dry run reads `db.json`, a saved library index if present, Greatest, and
|
||||
|
||||
+2
-2
@@ -150,7 +150,7 @@ Everything is plain data, and the whole index lives in memory.
|
||||
|
||||
The reader lays pages out before they load, so it needs every image's size. Instead of decoding images, the scanner reads the **first 32 KB** and parses the header; only if the size isn't there yet does it read on to 256 KB (and, for JPEG, the whole file). Parsing a prefix either finds nothing or the same size as the whole file, so the small first read never changes a result.
|
||||
|
||||
Uncached chapter folders and their images are probed **`SCAN_CONCURRENCY` at a time** (default 8, 1–64; chapters of a book in parallel, then assembled in folder order). A first scan over a NAS is bound by per-file latency, not bandwidth: on the server against a Synology (NFS) the old sequential 256 KB probe took about 20 ms per image; see §3.1 for the measurements. `SCAN_CONCURRENCY=1` restores the sequential order.
|
||||
Uncached chapter folders and their images are probed **`SCAN_CONCURRENCY` at a time** (default 8, 1–64; chapters of a book in parallel, then assembled in folder order). A first scan over a NAS is bound by per-file latency, not bandwidth: on a VM reading a Synology NAS over NFS the old sequential 256 KB probe took about 20 ms per image; see §3.1 for the measurements. `SCAN_CONCURRENCY=1` restores the sequential order.
|
||||
|
||||
| Format | Where the size is |
|
||||
|--------|-------------------|
|
||||
@@ -163,7 +163,7 @@ Uncached chapter folders and their images are probed **`SCAN_CONCURRENCY` at a t
|
||||
|
||||
### 3.1 Measured first-scan speed
|
||||
|
||||
the server (Debian VM, 4 vCPU) reading a Synology over NFS 4.0, page cache dropped before each run, production
|
||||
A Debian VM (4 vCPU) reading a Synology NAS over NFS 4.0, page cache dropped before each run, production
|
||||
running alongside. Each variant scanned a different manga of ~5,000 pages (the NAS caches what it has just read).
|
||||
|
||||
| Variant | ms per image (round 1 / round 2) |
|
||||
|
||||
+14
-16
@@ -41,24 +41,24 @@ separate lifetimes.
|
||||
The optional `peer-gateway` Compose profile publishes port 8093 on **127.0.0.1** by
|
||||
default. It uses the same image as reader/scraper, a read-only library mount, a
|
||||
read-only peering policy/identity mount, and its own writable usage-counter mount.
|
||||
The gateway needs an external-facing Docker network for its host port to work on
|
||||
Docker Desktop; it also joins the private backend. It has no reader account database,
|
||||
The gateway needs an external-facing Docker network for its host port to be
|
||||
published; it also joins the private backend. It has no reader account database,
|
||||
scraper queue, Docker socket, or writable library mount. It must never be scaled
|
||||
past one instance for the same usage-counter directory.
|
||||
|
||||
1. Build the image and start the normal stack as documented in
|
||||
[`compose-deployment.md`](compose-deployment.md). Set the local server name and
|
||||
[`deployment.md`](deployment.md). Set the local server name and
|
||||
generate its certificate in **Settings → Admin → Peering**. Keep
|
||||
`/data/peering/identity.key` private and backed up.
|
||||
2. Start the optional service with `docker compose --profile peering up -d
|
||||
peer-gateway`. Verify `docker compose --profile peering ps` shows healthy and
|
||||
`https://127.0.0.1:8093/health` responds. `/health` is deliberately unauthenticated
|
||||
and contains only `{ "ok": true }`.
|
||||
3. Give the other server a reachable HTTPS endpoint for port 8093. A private
|
||||
Tailscale TCP forward is recommended for the pilot; the default loopback binding
|
||||
is not remotely reachable. Do not expose port 8093 on the public internet merely
|
||||
to make pairing work. Use a stable tailnet DNS name, not a changing IP. Keep
|
||||
the scraper (8091) private.
|
||||
3. Give the other server a reachable HTTPS endpoint for port 8093. A private VPN
|
||||
(WireGuard, Tailscale or similar) with a TCP forward is recommended; the default
|
||||
loopback binding is not remotely reachable. Do not expose port 8093 on the public
|
||||
internet merely to make pairing work. Use a stable DNS name, not a changing IP.
|
||||
Keep the scraper (8091) private.
|
||||
|
||||
The `PEER_BIND`, `PEER_PORT`, `PEER_DATA_PATH`, and `PEER_USAGE_PATH` Compose variables
|
||||
control the bind and persistent paths. The gateway's internal port remains 8093.
|
||||
@@ -132,9 +132,10 @@ not shared. Catalog and chapter lists are short-lived in-memory caches. Turning
|
||||
serving blocks subsequent requests immediately, though an already streaming image
|
||||
may finish its current chunk/response.
|
||||
|
||||
The initial implementation sends original image bytes; it does not transcode them.
|
||||
See [`vision/server-peering-compression-test.md`](../vision/server-peering-compression-test.md)
|
||||
for the measured reason. The transport accepts a small amount of controlled page
|
||||
The initial implementation sends original image bytes; it does not transcode them:
|
||||
lossless compression of already encoded pages saved only about 6% of the transfer size
|
||||
in a measurement over 287 pages (webtoon strips about 11%, PNG chapters under 1%);
|
||||
`node tools/benchmark-peer-transfer.js` repeats it on your own library. The transport accepts a small amount of controlled page
|
||||
parallelism, not unbounded connections.
|
||||
|
||||
## Tests and safe operation
|
||||
@@ -149,11 +150,8 @@ checks the queue remains paused, and tears both stacks down. The UI suite checks
|
||||
guided setup, approvals, and non-admin boundary. An isolated single-stack Compose
|
||||
acceptance is:
|
||||
|
||||
```powershell
|
||||
$env:APP_IMAGE = 'mangacircus:local' # replace with the image just built
|
||||
$env:QA_PORT = '8188'
|
||||
$env:QA_PEER_PORT = '8193'
|
||||
node tools/test-compose.js --peer-gateway --peer-ui
|
||||
```bash
|
||||
APP_IMAGE=mangacircus:local QA_PORT=8188 QA_PEER_PORT=8193 node tools/test-compose.js --peer-gateway --peer-ui
|
||||
```
|
||||
|
||||
It makes disposable library/data, four containers, a certificate and a browser
|
||||
|
||||
+3
-3
@@ -15,7 +15,7 @@ The reader is the only writer of users/progress and the catalog. The scraper is
|
||||
only queue owner and content publisher, and it records its own failures and incidents.
|
||||
Each service's supervisor owns only that child process. The coding agent is an
|
||||
**escalation/maintenance operator**, not a necessary part of normal downloading.
|
||||
The recurring Codex check is an additional backstop, not the runtime watchdog.
|
||||
A periodic check by an operator or coding agent is an additional backstop, not the runtime watchdog.
|
||||
|
||||
## Automatic recovery implemented
|
||||
|
||||
@@ -89,8 +89,8 @@ Storage:
|
||||
- `data/scraper/supervisor.lock`, `daemon.lock`: native process ownership; transient.
|
||||
|
||||
Scraper and supervisor emit structured logs. The queue history retains
|
||||
200 incidents; the UI shows a bounded subset. Docker logs are rotated. Native Windows
|
||||
supervisor logs remain append-only: archive them during stopped maintenance if needed.
|
||||
200 incidents; the UI shows a bounded subset. Docker logs are rotated. Native supervisor logs
|
||||
(`tools/run-supervisor.ps1`) remain append-only: archive them during stopped maintenance if needed.
|
||||
Never export user databases, tokens, cookies, signed query strings or unredacted environments.
|
||||
|
||||
## What self-repair intentionally does not mean
|
||||
|
||||
@@ -31,11 +31,9 @@ npm start # reader only
|
||||
npm run start:scraper # daemon only
|
||||
```
|
||||
|
||||
These commands are for native development only. The current Windows installation
|
||||
runs the Docker Compose stack; the old Scheduled Task is disabled.
|
||||
Never start a native supervisor or daemon against its mounted state. Its observed
|
||||
ports, logs, persisted settings, restart procedure, and recovery snapshot are recorded in
|
||||
[`local-deployment.md`](local-deployment.md).
|
||||
These commands are for native development. Deployments run the Docker Compose stack
|
||||
([`deployment.md`](deployment.md)); never start a native supervisor or daemon against
|
||||
the state a container has mounted.
|
||||
|
||||
| Variable | Default | Purpose |
|
||||
|---|---|---|
|
||||
|
||||
@@ -73,8 +73,8 @@ not an editor for Docker environment variables or an unrestricted restart consol
|
||||
`npm test` covers route defaults, deep links, ownership and existing service/API
|
||||
behavior. `node tools/test-compose.js --settings-ui` adds optional Playwright
|
||||
headless browser QA against the disposable Compose fixture, never the real user
|
||||
database or queue. Make Playwright available on the development machine; Windows
|
||||
uses installed headless Edge by default, overridable with `QA_BROWSER_CHANNEL`.
|
||||
database or queue. Make Playwright available on the development machine; on Windows it
|
||||
uses the installed headless Edge by default, overridable with `QA_BROWSER_CHANNEL`.
|
||||
The application itself remains dependency-free.
|
||||
|
||||
The UI check verifies desktop and 390×844 mobile navigation, form saves, draft
|
||||
|
||||
Reference in new issue
Block a user