feat: show why units failed, with their last journal lines
Failed jobs keep their kind: units that broke or timed out get a failure panel entry with the reason systemd reports (exit status, signal, timeout, start limit, status text) and their last journal lines read through libsystemd. Units that failed only through a dependency are dimmed and left out of the panel, and units that crash after starting count as failed. Restructure the repository around this: - move the library and binary into bootvine/ and the scripted demos into their own unpublished scripted-demos crate, with their VHS tapes - move the crane builds and NixOS module into nix/ - group each sample VM with its check under vm-tests/, adding text-console, scripted-initrd, encrypted-volume and failures machines and a vm-failures check - add docs/guide.md walking through the screen with every recording, keep one recording in the README, and re-render all recordings
This commit is contained in:
1 parent
f2544a964a
commit
7e95cbd8e9
96 files changed
+2027
-462
No files matched your search
+44
-1
@@ -11,6 +11,8 @@ while it does. None of this is needed to install or configure it.
|
||||
PID 1 over systemd's private socket, which
|
||||
[zbus](https://github.com/dbus2/zbus) cannot use; see
|
||||
[systemd's private socket](#systemds-private-socket).
|
||||
- **libsystemd.** It reads the last journal lines of a failed unit; see
|
||||
[failures](#failures).
|
||||
- **A virtual terminal.** It draws on `tty12` and switches the console to it.
|
||||
|
||||
## Overview
|
||||
@@ -116,7 +118,7 @@ From then on it follows the signals:
|
||||
| Signal | Carries | Used for |
|
||||
| --- | --- | --- |
|
||||
| `JobNew` | job id, job path, unit name | a row in the tree |
|
||||
| `JobRemoved` | job id, result | the row finishes as done, failed or skipped |
|
||||
| `JobRemoved` | job id, unit name, result | the row finishes as done, failed or skipped |
|
||||
| `PropertiesChanged` on a unit | `ActiveState` and its timestamps | running since, and the timeline |
|
||||
| `StartupFinished` | | the boot reached its default target |
|
||||
|
||||
@@ -151,6 +153,47 @@ systemd's `…TimestampMonotonic` properties count microseconds of
|
||||
clock, so the times it shows are measured from the kernel start, even for
|
||||
units that ran before bootvine did.
|
||||
|
||||
### Failures
|
||||
|
||||
`JobRemoved` says how a job ended: `failed`, `timeout` or `dependency`. Like
|
||||
systemd's own `[FAILED]`, `[ TIME ]` and `[DEPEND]`, bootvine tells them
|
||||
apart: a unit that failed only because a dependency did is shown dimmed in the
|
||||
tree and left out of the failure panel, so the panel points at the unit that
|
||||
actually broke. A unit whose `ActiveState` turns `failed` without a failed job,
|
||||
such as a service crashing after it started, counts as failed too.
|
||||
|
||||
For a unit that broke or timed out, bootvine collects the cause once:
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant PID1 as systemd (PID 1)
|
||||
participant Bootvine as bootvine
|
||||
participant Journal as journal files
|
||||
|
||||
PID1-->>Bootvine: ActiveState=failed, or JobRemoved failed/timeout
|
||||
Bootvine->>PID1: GetUnit, Properties.GetAll on the unit type
|
||||
PID1->>Bootvine: Result, ExecMainCode, ExecMainStatus, StatusText
|
||||
Note over Bootvine: wait 200 ms for journald, longer while nothing is there yet
|
||||
Bootvine->>Journal: sd_journal, matching the unit in this boot
|
||||
Journal->>Bootvine: last messages, newest last
|
||||
Note over Bootvine: reason and lines in the failure panel
|
||||
```
|
||||
|
||||
- **The reason** comes from the unit type's `Result` (`exit-code`, `signal`,
|
||||
`timeout`, `start-limit-hit`, `oom-kill`, …), with the exit status or signal
|
||||
of the main process, and the service's `StatusText` when it set one. These
|
||||
are read when the unit fails, since they change at that moment.
|
||||
- **The journal lines** match the unit the same way `journalctl -u` does
|
||||
(`_SYSTEMD_UNIT=` or `UNIT=`, in the current boot), without PID 1's own
|
||||
"Starting…", "Failed with result…" and "Failed to start…" lines, which the
|
||||
reason already covers. In the initrd they come from `/run/log/journal`. The
|
||||
journal is opened without `SD_JOURNAL_LOCAL_ONLY`: libsystemd caches the
|
||||
initrd's machine ID, which would hide the real root's files after the
|
||||
switch, and the boot ID match already keeps other machines out.
|
||||
- **libsystemd** is linked rather than calling `journalctl`, since dracut,
|
||||
mkinitcpio and the NixOS initrd builder copy the libraries a binary links
|
||||
into the initramfs along with it, while `journalctl` may not be there.
|
||||
|
||||
## Surviving the switch to the real root
|
||||
|
||||
At the end of the initrd, systemd mounts the real root at `/sysroot`, isolates
|
||||
|
||||
@@ -81,6 +81,14 @@ queued-bar = "gray"
|
||||
running = "yellow"
|
||||
running-label = "white"
|
||||
|
||||
[colors.failures]
|
||||
rule = "dark-gray"
|
||||
title = "light-red"
|
||||
unit = "light-red"
|
||||
reason = "white"
|
||||
log = "gray"
|
||||
overflow = "dark-gray"
|
||||
|
||||
[colors.footer]
|
||||
rule = "dark-gray"
|
||||
done = "green"
|
||||
@@ -114,6 +122,7 @@ Each `[colors.*]` table covers one part of the screen:
|
||||
| `stages` | the finished stages, such as the initrd |
|
||||
| `tree` | the unit tree |
|
||||
| `timeline` | the timeline of the slowest units |
|
||||
| `failures` | the failed units with their reason and last journal lines |
|
||||
| `footer` | the tallies of done, running, waiting and failed units |
|
||||
| `password` | the passphrase popup |
|
||||
|
||||
|
||||
+32
-9
@@ -1,5 +1,19 @@
|
||||
# Development
|
||||
|
||||
## Layout
|
||||
|
||||
| Folder | Holds |
|
||||
| --- | --- |
|
||||
| `bootvine/` | the published crate: the library, the `bootvine` binary and the unit and initrd files it installs |
|
||||
| `scripted-demos/` | the scripted boot, shutdown and passphrase demos, and the VHS tapes that record them |
|
||||
| `xtask/` | repository tasks run with `cargo xtask` |
|
||||
| `vm-tests/` | the sample NixOS machines, one folder each with its VM check |
|
||||
| `nix/` | the crane builds and the NixOS module |
|
||||
| `recordings/` | the animations and screenshots the README and the [guide](guide.md) show |
|
||||
|
||||
`cargo build` and `cargo test` act on every crate. `cargo run` needs the binary:
|
||||
`--bin bootvine`, `--bin scripted-boot` and so on.
|
||||
|
||||
## Setup
|
||||
|
||||
Work inside the dev shell (`direnv allow` once, with
|
||||
@@ -57,24 +71,32 @@ locally from the dev shell with `release-plz update`.
|
||||
| What | Command |
|
||||
| --- | --- |
|
||||
| Model and render tests | `cargo test --workspace` |
|
||||
| Scripted demos | `cargo run --example boot` (also `shutdown`, `luks`) |
|
||||
| Sample machine in a window | `nix run .#sample-machine-vm` |
|
||||
| Scripted demos | `cargo run --bin scripted-boot` (also `scripted-shutdown`, `scripted-luks-unlock`) |
|
||||
| Sample machines in a window | `nix run .#sample-vm-<machine>`, see the [guide](guide.md#sample-machines) |
|
||||
| Boot and shutdown in a VM | `nix build .#checks.x86_64-linux.vm-boot-and-shutdown` |
|
||||
| Passphrase prompt in a VM | `nix build .#checks.x86_64-linux.vm-luks-passphrase` |
|
||||
| Failure panel in a VM | `nix build .#checks.x86_64-linux.vm-failures` |
|
||||
|
||||
To watch your own user manager start the sample units:
|
||||
|
||||
```sh
|
||||
units=$(nix build --no-link --print-out-paths .#sample-unit-files)
|
||||
systemctl --user link --runtime $units/*
|
||||
systemctl --user start --no-block bootvine-demo.target
|
||||
cargo run -- --user --target bootvine-demo.target --hold 3s
|
||||
systemctl --user start --no-block bootvine-sample-services.target
|
||||
cargo run --bin bootvine -- --user --target bootvine-sample-services.target --hold 3s
|
||||
systemctl --user stop $(ls $units)
|
||||
systemctl --user disable --runtime $(ls $units)
|
||||
```
|
||||
|
||||
Everything a sample machine needs sits in its own folder under `vm-tests/`:
|
||||
`machine.nix`, and for the machines with a check, `check.nix` and `check.py`.
|
||||
`vm-tests/shared/` holds the settings every machine shares, the window setup
|
||||
for `nix run` and the console helpers for the checks. A new machine is a new
|
||||
folder plus its name in `vm-tests/default.nix`.
|
||||
|
||||
The VM checks read the console as text, hold the boot at fixed points and wait
|
||||
on marker files instead of sleeping. To step through one by hand, run
|
||||
on marker files instead of sleeping. The failures check starts each failing
|
||||
unit by hand, so it knows which one is newest in the panel. To step through one by hand, run
|
||||
`nix run .#checks.x86_64-linux.vm-boot-and-shutdown.driverInteractive` and
|
||||
call `test_script()`.
|
||||
|
||||
@@ -83,12 +105,13 @@ exports libraries that clash with [QEMU](https://www.qemu.org) or Chromium.
|
||||
|
||||
## Repository tasks
|
||||
|
||||
- `nix run .#record-screencasts` re-records the README animations from the
|
||||
scripted demos and copies the passphrase screenshot from the VM check.
|
||||
- `nix run .#record-screencasts` re-records the animations from the scripted
|
||||
demos and copies the passphrase and failures screenshots from the VM checks
|
||||
into `recordings/`, for the README and the [guide](guide.md).
|
||||
- `cargo xtask sync-default-config` rewrites the defaults block in
|
||||
[configuration](configuration.md#defaults) from `bootvine defaults`.
|
||||
- `cargo run -- install --destdir <dir> --initrd dracut` stages an install
|
||||
without touching the system; `cargo run -- uninstall --destdir <dir>` removes
|
||||
- `cargo run --bin bootvine -- install --destdir <dir> --initrd dracut` stages an install
|
||||
without touching the system; `cargo run --bin bootvine -- uninstall --destdir <dir>` removes
|
||||
it again.
|
||||
|
||||
## Code style and design
|
||||
|
||||
+157
@@ -0,0 +1,157 @@
|
||||
# Guide
|
||||
|
||||
What bootvine shows on the console, part by part, with the recordings that go
|
||||
with each. The animations come from the [scripted demos](#scripted-demos); the
|
||||
screenshots come from the [VM checks](development.md#testing) on real NixOS
|
||||
machines.
|
||||
|
||||
## Reading the screen
|
||||
|
||||
<img src="../recordings/boot.gif" width="691" alt="A boot through the initrd into stage 2, with one failed unit">
|
||||
|
||||
From top to bottom:
|
||||
|
||||
1. **The header**: the host name, the system and its generation on NixOS
|
||||
(otherwise the `PRETTY_NAME` from `os-release`), and the kernel. Once a
|
||||
stage has finished, the total time since the kernel started sits on the
|
||||
right.
|
||||
2. **The stages**: each finished stage, such as `Initrd`, with how long it
|
||||
took, how many units finished in it and how many of them failed. The stage
|
||||
still running is the last line, ending in `Running`.
|
||||
3. **The tree** of units with a pending job. Each unit is nested under the unit
|
||||
waiting on it, and the milestone targets (`sysinit.target`, `basic.target`,
|
||||
`multi-user.target`, `graphical.target`) open the levels:
|
||||
- `◆` is the stage being worked on, `◇` one still to come, `√` one reached
|
||||
(`*`, `o` and `+` on console fonts without those glyphs);
|
||||
- `[|]` with a spinner is running, `[ ]` is waiting, `[√]` finished in the
|
||||
last couple of seconds (`keep-finished-units-for`);
|
||||
- `[×]` in bold red broke or timed out; dimmed red failed only because a
|
||||
unit it needed did;
|
||||
- the `Running` column counts up while a unit runs, and after 5 seconds the
|
||||
`Timeout` column shows the limit systemd will stop it at, red once it
|
||||
is passed;
|
||||
- a unit waiting on exactly one other, which waits on exactly one other,
|
||||
folds into one row joined by `─`; mounts and devices show their
|
||||
description instead of their unit name.
|
||||
|
||||
When the tree is taller than the screen, finished and waiting rows give way
|
||||
first, and a last line counts the units left out.
|
||||
4. **The failure panel**, only once a unit broke or timed out; see
|
||||
[failures](#failures).
|
||||
5. **The timeline**, in the room left below the tree: the milestones reached,
|
||||
with how many units they took, the slowest finished units (up to five, and
|
||||
none shorter than `hide-runs-shorter-than`) and the units still running,
|
||||
as bars against seconds since the kernel started. A `│` marks the switch
|
||||
from the initrd to the real root.
|
||||
6. **The footer**: the units done, running, waiting and failed so far.
|
||||
|
||||
## Booting through the initrd
|
||||
|
||||
The same recording shows a boot from the start. With a systemd initrd,
|
||||
bootvine starts about a second after the kernel:
|
||||
|
||||
- **The initrd stage**: udev settling, the root device and its file systems
|
||||
under `initrd.target`.
|
||||
- **The switch to the real root**: the initrd moves into the stages at the
|
||||
top, with its time and unit count, and the tree starts again under the
|
||||
default target. The timeline keeps the initrd's runs, left of the `│`.
|
||||
- **Stage 2** up to `graphical.target`.
|
||||
- **The handover**: bootvine hands the console back once the default target is
|
||||
reached, or earlier, once the display manager has switched to its own
|
||||
terminal.
|
||||
|
||||
How bootvine gets in that early and survives the switch is in
|
||||
[architecture](architecture.md#starting-in-the-initrd).
|
||||
|
||||
## Shutdown
|
||||
|
||||
<img src="../recordings/shutdown.gif" width="691" alt="A poweroff: units stopping under the targets waiting on them">
|
||||
|
||||
On a poweroff, reboot or halt the tree turns around:
|
||||
|
||||
- its root is the target being reached, such as `poweroff.target`;
|
||||
- each unit being stopped sits under the units that have to stop before it,
|
||||
so the unit holding up the shutdown is at the bottom of its branch;
|
||||
- `shutdown.target`, `umount.target` and `final.target` open the levels;
|
||||
- the screen stays until systemd hands over to `systemd-shutdown`.
|
||||
|
||||
## Passphrase prompts
|
||||
|
||||
<img src="../recordings/passphrase.png" width="640" alt="A LUKS passphrase prompt floating over the initrd tree">
|
||||
|
||||
When a unit asks for a passphrase, such as `systemd-cryptsetup@….service`
|
||||
unlocking a [LUKS](https://gitlab.com/cryptsetup/cryptsetup) volume, the
|
||||
question floats over a dimmed tree:
|
||||
|
||||
- typing shows one mark per character;
|
||||
- `Enter` sends it, and the popup says it is checking;
|
||||
- a wrong passphrase brings the prompt back with "Wrong passphrase, try
|
||||
again";
|
||||
- once unlocked, the popup closes and the boot carries on.
|
||||
|
||||
If another password agent, such as Plymouth's, was already asking when
|
||||
bootvine started, bootvine waits behind it.
|
||||
|
||||
## Failures
|
||||
|
||||
<img src="../recordings/failures.png" width="640" alt="The failure panel listing failed units with their reason and last journal lines">
|
||||
|
||||
A unit that broke or timed out gets an entry in the failure panel, newest
|
||||
first:
|
||||
|
||||
- **The headline**: the unit and why it failed, such as `exited with status 2`,
|
||||
`killed by SIGSEGV`, `timed out`, `restarted too often` or
|
||||
`exited with status 203` (systemd could not run the command). A service that
|
||||
reported a status with `sd_notify` adds it, as in
|
||||
`exited with status 1: Cannot reach database`.
|
||||
- **The journal lines**: up to six of the unit's last messages from this boot,
|
||||
the same ones `journalctl -u` shows, without systemd's own "Starting…" and
|
||||
"Failed with result…" lines.
|
||||
- **Out of room**: the newest failure keeps its lines, older ones shrink to
|
||||
their headline, and a last line counts the rest.
|
||||
- **What stays out**: a unit that failed only because one it needed did stays
|
||||
dimmed in the tree, so the panel points at the unit that actually broke.
|
||||
The footer counts both.
|
||||
|
||||
Failures from the initrd stay in the panel after the switch to the real root.
|
||||
The screenshot comes from the `failures` sample machine, with every kind of
|
||||
failure systemd reports.
|
||||
|
||||
## Trying it
|
||||
|
||||
Neither of these touches your system's boot.
|
||||
|
||||
### Scripted demos
|
||||
|
||||
Play a scripted boot, poweroff and passphrase prompt in your terminal, with
|
||||
[Rust](https://www.rust-lang.org/tools/install) installed:
|
||||
|
||||
```sh
|
||||
git clone https://codeberg.org/wvhulle/bootvine.git
|
||||
cd bootvine
|
||||
cargo run --bin scripted-boot
|
||||
cargo run --bin scripted-shutdown
|
||||
cargo run --bin scripted-luks-unlock
|
||||
```
|
||||
|
||||
`q` quits. The encrypted root unlocks with the passphrase `demo`; `Esc` quits
|
||||
while it asks. These are the scenarios the animations above were recorded
|
||||
from.
|
||||
|
||||
### Sample machines
|
||||
|
||||
Each sample machine boots in a [QEMU](https://www.qemu.org) window with
|
||||
[Nix](https://nixos.org/download/), and logs in as root at the end. `poweroff`
|
||||
shows the shutdown.
|
||||
|
||||
| Run | Shows |
|
||||
| --- | --- |
|
||||
| `nix run .#sample-vm-slow-services` | slow units in the initrd and stage 2, one failing unit, and the handover to a greetd login |
|
||||
| `nix run .#sample-vm-text-console` | the same units without a display manager, handing over at the default target |
|
||||
| `nix run .#sample-vm-scripted-initrd` | the same units with NixOS's scripted initrd, so bootvine starts in stage 2 only |
|
||||
| `nix run .#sample-vm-encrypted-volume` | a LUKS volume unlocked in the initrd; the passphrase is `sample passphrase` |
|
||||
| `nix run .#sample-vm-failures` | a firmware failure in the initrd, then units that exit with an error, crash, time out, cannot be run, restart too often, report a status, die after starting, and one that fails only through its dependency |
|
||||
|
||||
Each machine keeps its disk in `~/.cache/bootvine-<machine>.qcow2`; delete it
|
||||
to start fresh. The machines live in `vm-tests/`, one folder each, next to the
|
||||
checks that drive them.
|
||||
+2
-2
@@ -48,7 +48,7 @@ as well, the initrd has to run systemd.
|
||||
1. Install the build dependencies and Rust:
|
||||
|
||||
```sh
|
||||
sudo apt install build-essential pkg-config libdbus-1-dev git curl
|
||||
sudo apt install build-essential pkg-config libdbus-1-dev libsystemd-dev git curl
|
||||
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
|
||||
```
|
||||
|
||||
@@ -88,7 +88,7 @@ as well, the initrd has to run systemd.
|
||||
## Other distributions
|
||||
|
||||
Install as in the Ubuntu steps, with your distribution's packages for a C
|
||||
toolchain, `pkg-config` and the libdbus headers.
|
||||
toolchain, `pkg-config` and the libdbus and libsystemd headers.
|
||||
|
||||
For the initrd, use the dracut module as above, or on
|
||||
[mkinitcpio](https://wiki.archlinux.org/title/Mkinitcpio):
|
||||
|
||||
Reference in new issue
Block a user