feat: show why units failed, with their last journal lines

Failed jobs keep their kind: units that broke or timed out get a failure
panel entry with the reason systemd reports (exit status, signal, timeout,
start limit, status text) and their last journal lines read through
libsystemd. Units that failed only through a dependency are dimmed and left
out of the panel, and units that crash after starting count as failed.

Restructure the repository around this:
- move the library and binary into bootvine/ and the scripted demos into
  their own unpublished scripted-demos crate, with their VHS tapes
- move the crane builds and NixOS module into nix/
- group each sample VM with its check under vm-tests/, adding text-console,
  scripted-initrd, encrypted-volume and failures machines and a vm-failures
  check
- add docs/guide.md walking through the screen with every recording, keep
  one recording in the README, and re-render all recordings
This commit is contained in:
Willem Vanhulle committed 2026-10-30 14:59:10 +01:00
1 parent f2544a964a
commit 7e95cbd8e9
96 files changed
+2027 -462

No files matched your search

+44 -1
View File
@@ -11,6 +11,8 @@ while it does. None of this is needed to install or configure it.
PID 1 over systemd's private socket, which
[zbus](https://github.com/dbus2/zbus) cannot use; see
[systemd's private socket](#systemds-private-socket).
- **libsystemd.** It reads the last journal lines of a failed unit; see
[failures](#failures).
- **A virtual terminal.** It draws on `tty12` and switches the console to it.
## Overview
@@ -116,7 +118,7 @@ From then on it follows the signals:
| Signal | Carries | Used for |
| --- | --- | --- |
| `JobNew` | job id, job path, unit name | a row in the tree |
| `JobRemoved` | job id, result | the row finishes as done, failed or skipped |
| `JobRemoved` | job id, unit name, result | the row finishes as done, failed or skipped |
| `PropertiesChanged` on a unit | `ActiveState` and its timestamps | running since, and the timeline |
| `StartupFinished` | | the boot reached its default target |
@@ -151,6 +153,47 @@ systemd's `…TimestampMonotonic` properties count microseconds of
clock, so the times it shows are measured from the kernel start, even for
units that ran before bootvine did.
### Failures
`JobRemoved` says how a job ended: `failed`, `timeout` or `dependency`. Like
systemd's own `[FAILED]`, `[ TIME ]` and `[DEPEND]`, bootvine tells them
apart: a unit that failed only because a dependency did is shown dimmed in the
tree and left out of the failure panel, so the panel points at the unit that
actually broke. A unit whose `ActiveState` turns `failed` without a failed job,
such as a service crashing after it started, counts as failed too.
For a unit that broke or timed out, bootvine collects the cause once:
```mermaid
sequenceDiagram
participant PID1 as systemd (PID 1)
participant Bootvine as bootvine
participant Journal as journal files
PID1-->>Bootvine: ActiveState=failed, or JobRemoved failed/timeout
Bootvine->>PID1: GetUnit, Properties.GetAll on the unit type
PID1->>Bootvine: Result, ExecMainCode, ExecMainStatus, StatusText
Note over Bootvine: wait 200 ms for journald, longer while nothing is there yet
Bootvine->>Journal: sd_journal, matching the unit in this boot
Journal->>Bootvine: last messages, newest last
Note over Bootvine: reason and lines in the failure panel
```
- **The reason** comes from the unit type's `Result` (`exit-code`, `signal`,
`timeout`, `start-limit-hit`, `oom-kill`, …), with the exit status or signal
of the main process, and the service's `StatusText` when it set one. These
are read when the unit fails, since they change at that moment.
- **The journal lines** match the unit the same way `journalctl -u` does
(`_SYSTEMD_UNIT=` or `UNIT=`, in the current boot), without PID 1's own
"Starting…", "Failed with result…" and "Failed to start…" lines, which the
reason already covers. In the initrd they come from `/run/log/journal`. The
journal is opened without `SD_JOURNAL_LOCAL_ONLY`: libsystemd caches the
initrd's machine ID, which would hide the real root's files after the
switch, and the boot ID match already keeps other machines out.
- **libsystemd** is linked rather than calling `journalctl`, since dracut,
mkinitcpio and the NixOS initrd builder copy the libraries a binary links
into the initramfs along with it, while `journalctl` may not be there.
## Surviving the switch to the real root
At the end of the initrd, systemd mounts the real root at `/sysroot`, isolates
+9
View File
@@ -81,6 +81,14 @@ queued-bar = "gray"
running = "yellow"
running-label = "white"
[colors.failures]
rule = "dark-gray"
title = "light-red"
unit = "light-red"
reason = "white"
log = "gray"
overflow = "dark-gray"
[colors.footer]
rule = "dark-gray"
done = "green"
@@ -114,6 +122,7 @@ Each `[colors.*]` table covers one part of the screen:
| `stages` | the finished stages, such as the initrd |
| `tree` | the unit tree |
| `timeline` | the timeline of the slowest units |
| `failures` | the failed units with their reason and last journal lines |
| `footer` | the tallies of done, running, waiting and failed units |
| `password` | the passphrase popup |
+32 -9
View File
@@ -1,5 +1,19 @@
# Development
## Layout
| Folder | Holds |
| --- | --- |
| `bootvine/` | the published crate: the library, the `bootvine` binary and the unit and initrd files it installs |
| `scripted-demos/` | the scripted boot, shutdown and passphrase demos, and the VHS tapes that record them |
| `xtask/` | repository tasks run with `cargo xtask` |
| `vm-tests/` | the sample NixOS machines, one folder each with its VM check |
| `nix/` | the crane builds and the NixOS module |
| `recordings/` | the animations and screenshots the README and the [guide](guide.md) show |
`cargo build` and `cargo test` act on every crate. `cargo run` needs the binary:
`--bin bootvine`, `--bin scripted-boot` and so on.
## Setup
Work inside the dev shell (`direnv allow` once, with
@@ -57,24 +71,32 @@ locally from the dev shell with `release-plz update`.
| What | Command |
| --- | --- |
| Model and render tests | `cargo test --workspace` |
| Scripted demos | `cargo run --example boot` (also `shutdown`, `luks`) |
| Sample machine in a window | `nix run .#sample-machine-vm` |
| Scripted demos | `cargo run --bin scripted-boot` (also `scripted-shutdown`, `scripted-luks-unlock`) |
| Sample machines in a window | `nix run .#sample-vm-<machine>`, see the [guide](guide.md#sample-machines) |
| Boot and shutdown in a VM | `nix build .#checks.x86_64-linux.vm-boot-and-shutdown` |
| Passphrase prompt in a VM | `nix build .#checks.x86_64-linux.vm-luks-passphrase` |
| Failure panel in a VM | `nix build .#checks.x86_64-linux.vm-failures` |
To watch your own user manager start the sample units:
```sh
units=$(nix build --no-link --print-out-paths .#sample-unit-files)
systemctl --user link --runtime $units/*
systemctl --user start --no-block bootvine-demo.target
cargo run -- --user --target bootvine-demo.target --hold 3s
systemctl --user start --no-block bootvine-sample-services.target
cargo run --bin bootvine -- --user --target bootvine-sample-services.target --hold 3s
systemctl --user stop $(ls $units)
systemctl --user disable --runtime $(ls $units)
```
Everything a sample machine needs sits in its own folder under `vm-tests/`:
`machine.nix`, and for the machines with a check, `check.nix` and `check.py`.
`vm-tests/shared/` holds the settings every machine shares, the window setup
for `nix run` and the console helpers for the checks. A new machine is a new
folder plus its name in `vm-tests/default.nix`.
The VM checks read the console as text, hold the boot at fixed points and wait
on marker files instead of sleeping. To step through one by hand, run
on marker files instead of sleeping. The failures check starts each failing
unit by hand, so it knows which one is newest in the panel. To step through one by hand, run
`nix run .#checks.x86_64-linux.vm-boot-and-shutdown.driverInteractive` and
call `test_script()`.
@@ -83,12 +105,13 @@ exports libraries that clash with [QEMU](https://www.qemu.org) or Chromium.
## Repository tasks
- `nix run .#record-screencasts` re-records the README animations from the
scripted demos and copies the passphrase screenshot from the VM check.
- `nix run .#record-screencasts` re-records the animations from the scripted
demos and copies the passphrase and failures screenshots from the VM checks
into `recordings/`, for the README and the [guide](guide.md).
- `cargo xtask sync-default-config` rewrites the defaults block in
[configuration](configuration.md#defaults) from `bootvine defaults`.
- `cargo run -- install --destdir <dir> --initrd dracut` stages an install
without touching the system; `cargo run -- uninstall --destdir <dir>` removes
- `cargo run --bin bootvine -- install --destdir <dir> --initrd dracut` stages an install
without touching the system; `cargo run --bin bootvine -- uninstall --destdir <dir>` removes
it again.
## Code style and design
+157
View File
@@ -0,0 +1,157 @@
# Guide
What bootvine shows on the console, part by part, with the recordings that go
with each. The animations come from the [scripted demos](#scripted-demos); the
screenshots come from the [VM checks](development.md#testing) on real NixOS
machines.
## Reading the screen
<img src="../recordings/boot.gif" width="691" alt="A boot through the initrd into stage 2, with one failed unit">
From top to bottom:
1. **The header**: the host name, the system and its generation on NixOS
(otherwise the `PRETTY_NAME` from `os-release`), and the kernel. Once a
stage has finished, the total time since the kernel started sits on the
right.
2. **The stages**: each finished stage, such as `Initrd`, with how long it
took, how many units finished in it and how many of them failed. The stage
still running is the last line, ending in `Running`.
3. **The tree** of units with a pending job. Each unit is nested under the unit
waiting on it, and the milestone targets (`sysinit.target`, `basic.target`,
`multi-user.target`, `graphical.target`) open the levels:
- `◆` is the stage being worked on, `◇` one still to come, `√` one reached
(`*`, `o` and `+` on console fonts without those glyphs);
- `[|]` with a spinner is running, `[ ]` is waiting, `[√]` finished in the
last couple of seconds (`keep-finished-units-for`);
- `[×]` in bold red broke or timed out; dimmed red failed only because a
unit it needed did;
- the `Running` column counts up while a unit runs, and after 5 seconds the
`Timeout` column shows the limit systemd will stop it at, red once it
is passed;
- a unit waiting on exactly one other, which waits on exactly one other,
folds into one row joined by `─`; mounts and devices show their
description instead of their unit name.
When the tree is taller than the screen, finished and waiting rows give way
first, and a last line counts the units left out.
4. **The failure panel**, only once a unit broke or timed out; see
[failures](#failures).
5. **The timeline**, in the room left below the tree: the milestones reached,
with how many units they took, the slowest finished units (up to five, and
none shorter than `hide-runs-shorter-than`) and the units still running,
as bars against seconds since the kernel started. A `│` marks the switch
from the initrd to the real root.
6. **The footer**: the units done, running, waiting and failed so far.
## Booting through the initrd
The same recording shows a boot from the start. With a systemd initrd,
bootvine starts about a second after the kernel:
- **The initrd stage**: udev settling, the root device and its file systems
under `initrd.target`.
- **The switch to the real root**: the initrd moves into the stages at the
top, with its time and unit count, and the tree starts again under the
default target. The timeline keeps the initrd's runs, left of the `│`.
- **Stage 2** up to `graphical.target`.
- **The handover**: bootvine hands the console back once the default target is
reached, or earlier, once the display manager has switched to its own
terminal.
How bootvine gets in that early and survives the switch is in
[architecture](architecture.md#starting-in-the-initrd).
## Shutdown
<img src="../recordings/shutdown.gif" width="691" alt="A poweroff: units stopping under the targets waiting on them">
On a poweroff, reboot or halt the tree turns around:
- its root is the target being reached, such as `poweroff.target`;
- each unit being stopped sits under the units that have to stop before it,
so the unit holding up the shutdown is at the bottom of its branch;
- `shutdown.target`, `umount.target` and `final.target` open the levels;
- the screen stays until systemd hands over to `systemd-shutdown`.
## Passphrase prompts
<img src="../recordings/passphrase.png" width="640" alt="A LUKS passphrase prompt floating over the initrd tree">
When a unit asks for a passphrase, such as `systemd-cryptsetup@….service`
unlocking a [LUKS](https://gitlab.com/cryptsetup/cryptsetup) volume, the
question floats over a dimmed tree:
- typing shows one mark per character;
- `Enter` sends it, and the popup says it is checking;
- a wrong passphrase brings the prompt back with "Wrong passphrase, try
again";
- once unlocked, the popup closes and the boot carries on.
If another password agent, such as Plymouth's, was already asking when
bootvine started, bootvine waits behind it.
## Failures
<img src="../recordings/failures.png" width="640" alt="The failure panel listing failed units with their reason and last journal lines">
A unit that broke or timed out gets an entry in the failure panel, newest
first:
- **The headline**: the unit and why it failed, such as `exited with status 2`,
`killed by SIGSEGV`, `timed out`, `restarted too often` or
`exited with status 203` (systemd could not run the command). A service that
reported a status with `sd_notify` adds it, as in
`exited with status 1: Cannot reach database`.
- **The journal lines**: up to six of the unit's last messages from this boot,
the same ones `journalctl -u` shows, without systemd's own "Starting…" and
"Failed with result…" lines.
- **Out of room**: the newest failure keeps its lines, older ones shrink to
their headline, and a last line counts the rest.
- **What stays out**: a unit that failed only because one it needed did stays
dimmed in the tree, so the panel points at the unit that actually broke.
The footer counts both.
Failures from the initrd stay in the panel after the switch to the real root.
The screenshot comes from the `failures` sample machine, with every kind of
failure systemd reports.
## Trying it
Neither of these touches your system's boot.
### Scripted demos
Play a scripted boot, poweroff and passphrase prompt in your terminal, with
[Rust](https://www.rust-lang.org/tools/install) installed:
```sh
git clone https://codeberg.org/wvhulle/bootvine.git
cd bootvine
cargo run --bin scripted-boot
cargo run --bin scripted-shutdown
cargo run --bin scripted-luks-unlock
```
`q` quits. The encrypted root unlocks with the passphrase `demo`; `Esc` quits
while it asks. These are the scenarios the animations above were recorded
from.
### Sample machines
Each sample machine boots in a [QEMU](https://www.qemu.org) window with
[Nix](https://nixos.org/download/), and logs in as root at the end. `poweroff`
shows the shutdown.
| Run | Shows |
| --- | --- |
| `nix run .#sample-vm-slow-services` | slow units in the initrd and stage 2, one failing unit, and the handover to a greetd login |
| `nix run .#sample-vm-text-console` | the same units without a display manager, handing over at the default target |
| `nix run .#sample-vm-scripted-initrd` | the same units with NixOS's scripted initrd, so bootvine starts in stage 2 only |
| `nix run .#sample-vm-encrypted-volume` | a LUKS volume unlocked in the initrd; the passphrase is `sample passphrase` |
| `nix run .#sample-vm-failures` | a firmware failure in the initrd, then units that exit with an error, crash, time out, cannot be run, restart too often, report a status, die after starting, and one that fails only through its dependency |
Each machine keeps its disk in `~/.cache/bootvine-<machine>.qcow2`; delete it
to start fresh. The machines live in `vm-tests/`, one folder each, next to the
checks that drive them.
+2 -2
View File
@@ -48,7 +48,7 @@ as well, the initrd has to run systemd.
1. Install the build dependencies and Rust:
```sh
sudo apt install build-essential pkg-config libdbus-1-dev git curl
sudo apt install build-essential pkg-config libdbus-1-dev libsystemd-dev git curl
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
```
@@ -88,7 +88,7 @@ as well, the initrd has to run systemd.
## Other distributions
Install as in the Ubuntu steps, with your distribution's packages for a C
toolchain, `pkg-config` and the libdbus headers.
toolchain, `pkg-config` and the libdbus and libsystemd headers.
For the initrd, use the dracut module as above, or on
[mkinitcpio](https://wiki.archlinux.org/title/Mkinitcpio):