feat: show why units failed, with their last journal lines
Failed jobs keep their kind: units that broke or timed out get a failure panel entry with the reason systemd reports (exit status, signal, timeout, start limit, status text) and their last journal lines read through libsystemd. Units that failed only through a dependency are dimmed and left out of the panel, and units that crash after starting count as failed. Restructure the repository around this: - move the library and binary into bootvine/ and the scripted demos into their own unpublished scripted-demos crate, with their VHS tapes - move the crane builds and NixOS module into nix/ - group each sample VM with its check under vm-tests/, adding text-console, scripted-initrd, encrypted-volume and failures machines and a vm-failures check - add docs/guide.md walking through the screen with every recording, keep one recording in the README, and re-render all recordings
This commit is contained in:
1 parent
f2544a964a
commit
7e95cbd8e9
96 files changed
+2027
-462
No files matched your search
+157
@@ -0,0 +1,157 @@
|
||||
# Guide
|
||||
|
||||
What bootvine shows on the console, part by part, with the recordings that go
|
||||
with each. The animations come from the [scripted demos](#scripted-demos); the
|
||||
screenshots come from the [VM checks](development.md#testing) on real NixOS
|
||||
machines.
|
||||
|
||||
## Reading the screen
|
||||
|
||||
<img src="../recordings/boot.gif" width="691" alt="A boot through the initrd into stage 2, with one failed unit">
|
||||
|
||||
From top to bottom:
|
||||
|
||||
1. **The header**: the host name, the system and its generation on NixOS
|
||||
(otherwise the `PRETTY_NAME` from `os-release`), and the kernel. Once a
|
||||
stage has finished, the total time since the kernel started sits on the
|
||||
right.
|
||||
2. **The stages**: each finished stage, such as `Initrd`, with how long it
|
||||
took, how many units finished in it and how many of them failed. The stage
|
||||
still running is the last line, ending in `Running`.
|
||||
3. **The tree** of units with a pending job. Each unit is nested under the unit
|
||||
waiting on it, and the milestone targets (`sysinit.target`, `basic.target`,
|
||||
`multi-user.target`, `graphical.target`) open the levels:
|
||||
- `◆` is the stage being worked on, `◇` one still to come, `√` one reached
|
||||
(`*`, `o` and `+` on console fonts without those glyphs);
|
||||
- `[|]` with a spinner is running, `[ ]` is waiting, `[√]` finished in the
|
||||
last couple of seconds (`keep-finished-units-for`);
|
||||
- `[×]` in bold red broke or timed out; dimmed red failed only because a
|
||||
unit it needed did;
|
||||
- the `Running` column counts up while a unit runs, and after 5 seconds the
|
||||
`Timeout` column shows the limit systemd will stop it at, red once it
|
||||
is passed;
|
||||
- a unit waiting on exactly one other, which waits on exactly one other,
|
||||
folds into one row joined by `─`; mounts and devices show their
|
||||
description instead of their unit name.
|
||||
|
||||
When the tree is taller than the screen, finished and waiting rows give way
|
||||
first, and a last line counts the units left out.
|
||||
4. **The failure panel**, only once a unit broke or timed out; see
|
||||
[failures](#failures).
|
||||
5. **The timeline**, in the room left below the tree: the milestones reached,
|
||||
with how many units they took, the slowest finished units (up to five, and
|
||||
none shorter than `hide-runs-shorter-than`) and the units still running,
|
||||
as bars against seconds since the kernel started. A `│` marks the switch
|
||||
from the initrd to the real root.
|
||||
6. **The footer**: the units done, running, waiting and failed so far.
|
||||
|
||||
## Booting through the initrd
|
||||
|
||||
The same recording shows a boot from the start. With a systemd initrd,
|
||||
bootvine starts about a second after the kernel:
|
||||
|
||||
- **The initrd stage**: udev settling, the root device and its file systems
|
||||
under `initrd.target`.
|
||||
- **The switch to the real root**: the initrd moves into the stages at the
|
||||
top, with its time and unit count, and the tree starts again under the
|
||||
default target. The timeline keeps the initrd's runs, left of the `│`.
|
||||
- **Stage 2** up to `graphical.target`.
|
||||
- **The handover**: bootvine hands the console back once the default target is
|
||||
reached, or earlier, once the display manager has switched to its own
|
||||
terminal.
|
||||
|
||||
How bootvine gets in that early and survives the switch is in
|
||||
[architecture](architecture.md#starting-in-the-initrd).
|
||||
|
||||
## Shutdown
|
||||
|
||||
<img src="../recordings/shutdown.gif" width="691" alt="A poweroff: units stopping under the targets waiting on them">
|
||||
|
||||
On a poweroff, reboot or halt the tree turns around:
|
||||
|
||||
- its root is the target being reached, such as `poweroff.target`;
|
||||
- each unit being stopped sits under the units that have to stop before it,
|
||||
so the unit holding up the shutdown is at the bottom of its branch;
|
||||
- `shutdown.target`, `umount.target` and `final.target` open the levels;
|
||||
- the screen stays until systemd hands over to `systemd-shutdown`.
|
||||
|
||||
## Passphrase prompts
|
||||
|
||||
<img src="../recordings/passphrase.png" width="640" alt="A LUKS passphrase prompt floating over the initrd tree">
|
||||
|
||||
When a unit asks for a passphrase, such as `systemd-cryptsetup@….service`
|
||||
unlocking a [LUKS](https://gitlab.com/cryptsetup/cryptsetup) volume, the
|
||||
question floats over a dimmed tree:
|
||||
|
||||
- typing shows one mark per character;
|
||||
- `Enter` sends it, and the popup says it is checking;
|
||||
- a wrong passphrase brings the prompt back with "Wrong passphrase, try
|
||||
again";
|
||||
- once unlocked, the popup closes and the boot carries on.
|
||||
|
||||
If another password agent, such as Plymouth's, was already asking when
|
||||
bootvine started, bootvine waits behind it.
|
||||
|
||||
## Failures
|
||||
|
||||
<img src="../recordings/failures.png" width="640" alt="The failure panel listing failed units with their reason and last journal lines">
|
||||
|
||||
A unit that broke or timed out gets an entry in the failure panel, newest
|
||||
first:
|
||||
|
||||
- **The headline**: the unit and why it failed, such as `exited with status 2`,
|
||||
`killed by SIGSEGV`, `timed out`, `restarted too often` or
|
||||
`exited with status 203` (systemd could not run the command). A service that
|
||||
reported a status with `sd_notify` adds it, as in
|
||||
`exited with status 1: Cannot reach database`.
|
||||
- **The journal lines**: up to six of the unit's last messages from this boot,
|
||||
the same ones `journalctl -u` shows, without systemd's own "Starting…" and
|
||||
"Failed with result…" lines.
|
||||
- **Out of room**: the newest failure keeps its lines, older ones shrink to
|
||||
their headline, and a last line counts the rest.
|
||||
- **What stays out**: a unit that failed only because one it needed did stays
|
||||
dimmed in the tree, so the panel points at the unit that actually broke.
|
||||
The footer counts both.
|
||||
|
||||
Failures from the initrd stay in the panel after the switch to the real root.
|
||||
The screenshot comes from the `failures` sample machine, with every kind of
|
||||
failure systemd reports.
|
||||
|
||||
## Trying it
|
||||
|
||||
Neither of these touches your system's boot.
|
||||
|
||||
### Scripted demos
|
||||
|
||||
Play a scripted boot, poweroff and passphrase prompt in your terminal, with
|
||||
[Rust](https://www.rust-lang.org/tools/install) installed:
|
||||
|
||||
```sh
|
||||
git clone https://codeberg.org/wvhulle/bootvine.git
|
||||
cd bootvine
|
||||
cargo run --bin scripted-boot
|
||||
cargo run --bin scripted-shutdown
|
||||
cargo run --bin scripted-luks-unlock
|
||||
```
|
||||
|
||||
`q` quits. The encrypted root unlocks with the passphrase `demo`; `Esc` quits
|
||||
while it asks. These are the scenarios the animations above were recorded
|
||||
from.
|
||||
|
||||
### Sample machines
|
||||
|
||||
Each sample machine boots in a [QEMU](https://www.qemu.org) window with
|
||||
[Nix](https://nixos.org/download/), and logs in as root at the end. `poweroff`
|
||||
shows the shutdown.
|
||||
|
||||
| Run | Shows |
|
||||
| --- | --- |
|
||||
| `nix run .#sample-vm-slow-services` | slow units in the initrd and stage 2, one failing unit, and the handover to a greetd login |
|
||||
| `nix run .#sample-vm-text-console` | the same units without a display manager, handing over at the default target |
|
||||
| `nix run .#sample-vm-scripted-initrd` | the same units with NixOS's scripted initrd, so bootvine starts in stage 2 only |
|
||||
| `nix run .#sample-vm-encrypted-volume` | a LUKS volume unlocked in the initrd; the passphrase is `sample passphrase` |
|
||||
| `nix run .#sample-vm-failures` | a firmware failure in the initrd, then units that exit with an error, crash, time out, cannot be run, restart too often, report a status, die after starting, and one that fails only through its dependency |
|
||||
|
||||
Each machine keeps its disk in `~/.cache/bootvine-<machine>.qcow2`; delete it
|
||||
to start fresh. The machines live in `vm-tests/`, one folder each, next to the
|
||||
checks that drive them.
|
||||
Reference in new issue
Block a user