NixOS on Jon Seager

The Immutable Linux Paradox

Mon, 01 Sep 2025 00:00:00 +0000

This article was originally posted on the Ubuntu Discourse, and is reposted here. I welcome comments and further discussion in that thread.

Immutable Linux distributions have been around since the early 2000s, but adoption has significantly accelerated in the last five years. Mainstream operating systems (OSes) such as macOS, Android, ChromeOS and iOS have all embraced similar principles, reflecting a growing trend toward resilience, longevity, and maintainability as core ideals of OS development.

Ubuntu Core has been at the forefront of this movement for IoT, appliances and edge deployments, with work ongoing to release a “Core Desktop” experience. Other projects such as NixOS, Fedora Silverblue and Red Hat image mode are gaining adoption, alongside more specialised immutable distributions such as SteamOS and Talos.

This post explores how different Linux distributions achieve immutability, the trade-offs, and why you should give it a try!

What is an immutable Linux distribution? #

The key principle of an immutable OS is that the core system is unchangeable at runtime.

Every OS installation has at least one filesystem that stores system software, user software, and user data. Immutable OSes must cleanly separate “system” and “user” software and data, such that regular user interactions cannot compromise the integrity of the OS.

Immutable deployments are often separated into three layers:

Base OS - immutable core, updated only through controlled mechanisms
Applications - user applications, often delivered in containerised formats such as Snap, Flatpak, AppImage, cpak
User data - writable and persistent, independent of OS updates or rollbacks

Immutable systems use atomic, transactional updates meaning updates are applied as unitary, indivisible operations that either wholly succeed, or fail completely and trigger an automated roll-back to a previous known-good state.

Why immutability? #

The major benefit of an immutable OS is resilience.

Immutable OSes make it easier to reproduce systems with a given configuration, which is particularly useful in scale-out use-cases such as cloud or IoT.

Traditional package managers often maintain a database of installed packages, consisting of those included in the base OS, and those explicitly installed by the user, and their dependencies. The package manager doesn’t have a clear notion of which packages make up the “core system”, and which are “optional”.

This can cause “configuration drift”, which occurs over time - a package could be explicitly installed by a user, used for a while and then removed, but without removing its dependencies. This leaves the system in a different, and somewhat undefined, state than it was in prior to the package being installed.

Often the traditional notion of OS security is improved with immutable OS concepts too. In most implementations, the core OS files are mounted read-only such that users cannot make changes - which also raises the bar for malicious modifications. When combined with technologies such as secure boot and confinement, immutable OSes can dramatically reduce the attack surface of a machine.

Finally, convenience! Immutable OSes often include recovery or rollback features, which enable users to “undo” a bad system change, reverting to a previous known-good revision.

The immutability paradox #

In reality, no general-purpose operating system is fully immutable.

There is always persistent, user-writable storage - because without this there would be a huge limitation on usefulness! Similarly, how can a system be truly immutable, yet still support software updates?

The terms “immutable” and “stateless” are often conflated - when in reality neither are excellent terms for describing what has become widely known as “immutable OSes”. This was explored in some depth in this blog post which proposes terms such as “image based” and “fully managed”.

By definition, changes to configuration, the installation of applications and the use of temporary runtime storage are all violations of immutability, and thus immutability concepts must be applied in some sort of layering system.

Striking the balance between ’true’ immutability and user experience is one of the hardest challenges in immutable OS design. A system that is too rigid can be difficult to manage and use, appearing inflexible to end users.

A common pattern is to run an immutable desktop OS and use virtualisation or containerisation technologies (e.g. LXD, Podman, toolbx Distrobox) to create mutable environments in which to work on projects. This results in a very stable workstation that benefits from immutability, with the flexibility of a traditional mutable OS where it’s needed.

Approaches to immutability #

Different distributions solve the immutability challenge in different ways. In this section we’ll explore the four different approaches of ostree based distributions, bootc based distributions, NixOS and Ubuntu Core.

Fedora Silverblue / CoreOS / EndlessOS (`ostree`) #

Fedora Silverblue and Fedora CoreOS are also popular choices for those exploring immutable OSes. The two share a lot of underlying technology with Silverblue targeting desktop use cases, and CoreOS targeting server deployments.

Both are based on ostree, which provides:

tools that combine a ‘git-like’ model for committing and downloading bootable filesystem trees, along with a layer for deploying them and managing the bootloader configuration.

Silverblue and CoreOS actually rely on rpm-ostree , a “hybrid image/package manager” which combines RPM packaging technology with ostree to manage deployments.

The update mechanism involves switching the filesystem to track a different remote “ref”, which is analogous to a git ref.

EndlessOS is based on Debian, but uses ostree to achieve immutability. EndlessOS is a desktop experience designed more for the “average user” and focuses on providing a reliable system that works well in low-bandwidth or offline situations.

Users often use Flatpak to install graphical user applications atop the immutable base, or a user-space package manager such as brew for other utilities.

ostree based distributions also support “package layering” which enables adding packages to the base system without fetching a whole new filesystem ref, but does require the system to be rebooted before the package is persistently available. The documentation notes that this approach is to be used “sparingly”, and that users should prefer using Flatpak or toolbx to access additional packages.

RHEL “Image Mode” (`bootc`) #

bootc based distributions use an alternate approach, packaging the base system into OCI containers (commonly referred to as Docker containers). Atomicity and transactionality are achieved by using container images to deliver the entire core system, and rebooting into a new revision.

RHEL Image Mode uses bootc. This technology capitalises on the success of OCI containers as a transport and delivery mechanism for software by packing an entire OS base image into a single container, including the kernel image.

The bootc project builds on ostree , but where ostree never delivered an opinionated “install mechanism”, bootc does. The contents of a bootc image is an ostree filesystem.

Installing new system packages generally means building a new base image, downloading that image and rebooting into it with a command such as bootc switch <image reference>.

Users often use Flatpak to install graphical user applications atop the immutable base, or a user-space package manager such as brew for other utilities.

NixOS #

The Nix project first appeared in 2003. NixOS is built on top of the Nix package manager, using it to manage both packages and system configuration.

NixOS defines the entire system through a declarative configuration, with changes applied via “generations” that can be rolled back. Changes to the system are applied by “rebuilding” the system configuration, which produces a new “generation”.

Nix packages, and therefore NixOS, eschews the traditional Unix FHS in favour of the Nix “store” and a collection of symlinks and wrappers managed by Nix. Only the Nix package manager can write to the store.

The Nix store also (mostly) enables the building and switching of generations without a reboot. Updates are atomic: new generations must build completely before they can be activated. The home-manager project extends these concepts to the user environment and dotfile management.

The impermanence project requires that every persistent directory is explicitly labelled, or else it’s deleted on every reboot, forcing the base OS to be rebuilt from the Nix store and system configuration - essentially “enforcing” core system immutability between reboots. This was inspired by blog posts “Erase Your Darlings” and “NixOS tmpfs as root”, which are worth a read, too!

Ubuntu Core #

Ubuntu Core achieves immutability by packaging every component (kernel, base system, applications) as Snaps.

Snap confinement enforces isolation, and snapd manages transactional updates and rollbacks. The system is designed for reliability, fleet management, and modular upgrades, making it well-suited for IoT and soon, desktop use.

The key components of an Ubuntu Core deployment are:

Gadget snap: provides boot assets, including board specific binaries and data (bootloader, device tree, etc.)
Kernel snap: kernel image and associated modules, along with initial ramdisk for system initialisation
Base snap: execution environment in which applications run - includes “base” Ubuntu LTS packages
System snaps: packages critical to system function such as Network-Manager, bluez, pulseaudio, etc.
Application snaps: define the functionality of the system, confined to a sandbox
Snapd: manages updates, rollbacks and snapshotting/restoring of user data

In a Core Desktop installation, the desktop environment (GNOME, Plasma, etc.), display manager, login manager would all be delivered as “system snaps”.

Snap confinement ensures packages cannot incorrectly interact with the underlying system or user data without explicit approval. In an Ubuntu Core deployment, this notion is extended to every component of the OS, offering a straightforward yet powerful way to manage risk for each system component.

Summary #

Immutable Linux distributions approach the immutability paradox differently. We explored four different approaches here, and you can learn more about other approaches taken by the likes of SUSE MicroOS (filesystem based immutability) and Vanilla OS (uses ABRoot) in this excellent blog post.

Ubuntu Core focuses on transactional packaging and a clean separation of system & user data. bootc-based systems take a full image-based approach, while NixOS offers extreme flexibility through declarative configuration, but at the cost of complexity.

If you’ve yet to try an immutable Linux distribution I’d recommend giving it a go. Whether you prioritise simplicity, security or declarative control there’s almost certainly an immutable Linux distribution that fits your needs.

From NixOS to Ubuntu

Tue, 24 Jun 2025 00:00:00 +0000

Following my appointment as VP Engineering for Ubuntu, I moved all of my machines from NixOS to Ubuntu. Being responsible for decisions that affect millions of Ubuntu users comes with, in my opinion, the obligation to use the product and live with those decisions myself.

Following years of running Arch Linux and NixOS, I imagined this would be uncomfortable, but was pleasantly surprised. In this post, I’ll outline my setup and a new philosophy for how I configure my machines.

Last year, I wrote in detail about my setup. Consider this post a “diff” on what’s changed since.

Recap: Why NixOS? #

NixOS affords a seemingly endless selection of applications, desktop environments and when combined with Home Manager it provides a consistent way to manage the configuration of almost all aspects of a system.

People have argued that this is overkill, and results in needlessly complex configurations that produce difficult to read error messages, and make a system more difficult to troubleshoot.

While I had some troubles adopting NixOS, I consistently felt that the positives outweighed the negatives: I loved being able to overlay individual packages and tweak fundamentals of the system in a machine specific way. I also liked being able to trivially reuse configuration for app and hardware configuration across my machines in a single flake, which remains (to my amazement) one of my most popular GitHub repositories.

As I planned my move to Ubuntu, I decided to change the way I used my computer to avoid fighting my machine, and any feeling that I could be missing out on the ~~complexity~~ flexibility I’d become so used to.

Adopting “JFUI” #

The guiding principle for my new way of thinking is JFUI: Just F*cking Use It!

The primary motivation behind JFUI is to pick applications for which the defaults are close enough to my preferences, then use them with as little (or no) configuration as possible.

By employing this principle, I should spend less of my time updating my configuration files as things change, and spend less time obsessing over every last theme detail for each application on my machine.

I’ve spent years collecting repositories full of dotfiles, which contains the ~2500 lines of configuration and scripts I used prior to moving to NixOS. As of today, the most recent commit in my Nix flake, cloc reports:

---------------------------------------------------
Language files blank comment code
---------------------------------------------------
Nix 113 401 232 3666
Bourne Again Shell 3 25 30 95
Bourne Shell 1 17 0 83
Markdown 1 19 1 81
YAML 4 8 3 51
diff 1 1 7 6
---------------------------------------------------
SUM: 123 471 273 3982
---------------------------------------------------

And that seems like a lot to me! Yet this number omits my Visual Studio Code configuration (a further 200 lines of JSON), various browser configurations, etc.

So I challenged myself to set up my Ubuntu machines with as little configuration as possible and to choose apps with better defaults.

Desktop Environment #

It’d been a long time since I’d used a computer regularly without a tiling window manager. I switched to sway in 2019, and to Hyprland in 2023.

I’ve built a lot of muscle memory by using a consistent keymap across both environments. I use a 57" ultrawide monitor, and the idea of using it without tiling seemed like total anarchy to me.

But it didn’t take me long to adapt to using GNOME again. Despite abstaining from it for years, I’ve always appreciated the visual design of GNOME and often used their apps as part of my tiling experience (Files, Papers, Loupe).

Features such as the keyring, light/dark mode switching worked for me under Sway/Hyprland, but it always required complex, fragile configuration. The out of the box experience for configuring WiFi networks and Bluetooth devices feels modern, and like a part of the OS, rather than a kit of parts. In my latter few months of running Hyprland, I struggled more with consistent theming and stability as Hyprland evolved separately from the themes and apps I liked.

I don’t attribute any blame for this; it’s a natural side effect of combining lots of independent and often complex parts from across the Linux desktop ecosystem, and I was consciously running pre 1.0 software knowing there would be issues because I enjoyed the overall experience.

GNOME Extensions #

This is the area where I violated JFUI the most, though some of it is temporary as I get further away from my existing workflow. I disable the Ubuntu dock, desktop icons, app indicators and Ubuntu tiling assistant. While the tiling assistant was a great step up on what came before it, it fell short of what I needed to manage windows on my large display.

I took inspiration from Omakub, which is where I discovered Tactile - a GNOME extension for tiling windows to a custom on-screen grid using the keyboard. I’ve found this to be invaluable, and easily the best window management experience for GNOME. I’ve customised the grid to the following ratios (using gsettings in a script):

I’ve grown to like Space Bar - it provides desktop workspaces akin to those in Sway/Hyprland which enabled me to use the muscle memory I’d developed over years. As I’ve progressed, I think I could live without Space Bar and use the native workspace features in GNOME, so I’ll experiment with that soon.

The final two are somewhat simpler: Emoji Copy (mapped to Super + E) and Clipboard History (mapped to Super + V), functions that were previously enabled by wofi with a couple of plugins and lots of configuration.

Editors #

I spend a lot of time in an editor. As I wrote last year, I’ve been using neovim and Visual Studio Code for some years. I never “managed” Visual Studio Code with NixOS because I always found their settings sync to be quite sufficient.

My nvim config wasn’t too complicated - though it is made to look simpler because Home Manager abstracts away the details of managing plugins. I hadn’t taken the time to set up language server support, code completion or other creature comforts that one might expect from an editor in 2025.

I’d dabbled before with Helix, a modal command-line editor not dissimilar from vim. Helix comes with a lot more out of the box: it supports the Language Server Protocol (LSP), has many built in colour schemes, supports fuzzy finding files/buffers, project wide search and more. The keymap took some practice, but my entire configuration amounts to 8 lines including some blank lines and provides many more modern features.

1
2
3
4
5
6
7
8
9


theme = "catppuccin_macchiato"

[editor]
cursorline = true
text-width = 100
rulers = [100]

[editor.lsp]
display-inlay-hints = true

There is a separate file with some LSP configuration, but only a few more lines. Once I got this set up, I spent about 8 weeks using just Helix to make sure that the keymap and operating model were sufficiently burned in to my mind. I’ve been really impressed - Helix is fast, the default features are great and I don’t anticipate returning to vim.

On the desktop side, I’d was intrigued by Zed. I don’t always feel the need for a desktop editor, but there are projects that I prefer working on in a more graphical environment. I last tried Zed about 18 months ago and found it a little too sparse on features, but things have really evolved since then and there is an encouraging rate of change on the project. I’ve been using it most days for the last 4-5 months, and while it still lacks some of the polish (and features) of Visual Studio Code, it’s significantly lighter on resources, and I’m much happier with the defaults (details below)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


{
 "ui_font_size": 18,
 "buffer_font_size": 18,
 "theme": {
 "mode": "system",
 "light": "Catppuccin Latte",
 "dark": "Catppuccin Macchiato"
 },
 "buffer_font_family": "MesloLGMDZ Nerd Font Mono",
 "ui_font_family": ".SystemUIFont",
 "base_keymap": "SublimeText",
}

Terminal #

I switched to zsh around 2013 and have used it every day since: starting with the infamous Oh My Zsh, powerlevel9k, and subsequently powerlevel10k, and finally settling on Starship with a couple of zsh plugins such as zsh-autosuggestions and zsh-syntax-highlighting.

My zsh config had become quite complex - an area in which NixOS/Home Manager really helped by providing (mostly)neat abstractions for plugins and starship integration, but the equivalent configuration on Ubuntu would have been 100s of lines of zsh, notwithstanding the need to load plugins in the right order, etc.

Most of the configuration I was doing with zsh was to imitate fish. I’d tried fish in the past, but reverted when I couldn’t use sudo !!, or /some/command $! and other tricks. I’d been following fish’s rewrite and decided to give it another go, and have stuck with it since. It fits my JFUI mantra perfectly, leaving me with just a few lines of config for essentially identical functionality:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


set fish_greeting ""

fish_add_path ~/.nix-profile/bin
fish_add_path ~/.local/bin
fish_add_path ~/.cargo/bin
fish_add_path ~/go/bin
fish_add_path ~/scripts

if status is-interactive
 starship init fish | source
 fzf --fish | source
 atuin init fish --disable-up-arrow | source
end

set -gx EDITOR hx
set -gx SUDO_EDITOR hx

I still occasionally fall foul of typing sudo !!, but overall I’ve found fish to be an excellent interactive shell replacement. In particular, the native tab completion support is leagues ahead of anything I ever managed to configure with zsh.

I moved from Alacritty to Ghostty, which has been absolutely excellent. It’s wicked fast, it has my favourite colour scheme built in, and I just love the way @mitchellh has set the project up for success in the long term. It also fits in nicely with my minimally configured applications, with excellent defaults out of the box.

1
2
3
4
5
6


theme = catppuccin-macchiato
font-family = MesloLGMDZ Nerd Font Mono
font-size = 14
window-padding-x = 10
window-padding-y = 10
window-decoration = false

Finally: tmux, which I had been happily using for many years, but slowly collecting configuration for. I decided to give Zellij a try, and haven’t looked back. It also has a hellishly complicated configuration in my case 😉:

1
2
3


theme "catppuccin-macchiato"
default_layout "compact"
show_startup_tips false

Software Availability #

I always felt incredibly spoiled by the vast availability of software for NixOS, and even more so that the contribution model was so simple that I was able to augment to collection myself.

On Ubuntu, I use Snaps wherever possible and fall back to archive and installing software with apt where it makes sense. If neither of those have what I need, I use the Nix package manager, which works well on Ubuntu. While some of the “additional” software might be installable using go install, or cargo install, or other package managers like brew, flakpak, npm, etc., I wanted to keep things as simple as possible, so if it’s not available from Ubuntu-native sources I get it from nixpkgs.

I’ve included what I’m actually getting from the Snap store (63 snaps), and from nixpkgs (21 packages) below.

Installed snaps

❯ snap list | tail -n+2
agendrr 0.1.1 3 latest/stable jnsgruk* -
astral-uv 0.7.13 627 latest/stable lengau classic
audacity 3.7.1 1208 latest/candidate snapcrafters* -
bare 1.0 5 latest/stable canonical** base
bitwarden 2025.5.1 140 latest/stable bitwarden** -
charmcraft 3.4.6 6672 latest/stable canonical** classic
code 18e3a1ec 197 latest/stable vscode** classic
core 16-2.61.4-20250508 17212 latest/stable canonical** core
core18 20250523 2887 latest/stable canonical** base
core20 20250526 2599 latest/stable canonical** base
core22 20250528 2010 latest/stable canonical** base
core24 20250526 1006 latest/stable canonical** base
desktop-security-center 0+git.f7ad73a 59 1/stable/… canonical** -
discord 0.0.98 243 latest/candidate snapcrafters* -
docker 28.1.1+1 3265 latest/stable canonical** -
dotrun 1.4.8 85 latest/stable canonicalwebteam -
ffmpeg-2404 7.1.1 75 latest/stable snapcrafters* -
firefox 139.0.4-1 6316 latest/stable/… mozilla** -
firmware-updater 0+git.22198be 167 1/stable/… canonical** -
ghstat 0.4.1 91 latest/stable jnsgruk* -
ght 1.11.7 110 latest/edge tbmb -
gimp 3.0.4 525 latest/stable snapcrafters* -
gnome-3-28-1804 3.28.0-19-g98f9e67.98f9e67 198 latest/stable canonical** -
gnome-3-34-1804 0+git.3556cb3 93 latest/stable canonical** -
gnome-42-2204 0+git.38ea591 202 latest/stable/… canonical** -
gnome-46-2404 0+git.d9f8bf6-sdk0+git.c8a281c 90 latest/stable canonical** -
go 1.24.4 10907 latest/stable canonical** classic
gopls 0.19.0 1089 latest/stable alexmurray* classic
goreleaser 2.10.2 1060 latest/stable caarlos0 classic
gtk-common-themes 0.1-81-g442e511 1535 latest/stable/… canonical** -
helix 25.01.1 91 latest/stable lauren-brock classic
icloudpd 1.28.1 12 latest/stable jnsgruk* -
jhack 0.4.4.0.13 461 latest/stable ppasotti -
jq 1.5+dfsg-1 6 latest/stable mvo* -
juju 3.6.7 31266 3/stable canonical** -
kubectl 1.33.2 3609 latest/stable canonical** classic
lxd 5.21.3-c5ae129 33110 5.21/stable canonical** -
mattermost-desktop 5.12.1 789 latest/stable snapcrafters* -
mesa-2404 24.2.8-snap183 887 latest/stable canonical** -
multipass 1.15.1 14535 latest/stable canonical** -
node 22.16.0 10226 22/stable iojs** classic
obsidian 1.8.10 47 latest/stable obsidianmd classic
pinta 3.0.1 56 latest/stable james-carroll* -
prompting-client 0+git.d542a5d 104 1/stable/… canonical** -
rambox 2.4.1 44 latest/stable ramboxapp** -
rockcraft 1.12.0 3367 latest/stable canonical** classic
ruff 0.11.13 1377 latest/stable lengau -
rustup 1.27.1 1471 latest/stable canonical** classic
shellcheck v0.10.0 1725 latest/stable koalaman -
shfmt 3.5.1 33 latest/stable ankushpathak -
signal-desktop 7.58.0 799 latest/candidate snapcrafters* -
snap-store 0+git.90575829 1270 2/stable/… canonical** -
snapcraft 8.9.4 15082 latest/stable canonical** classic
snapd 2.68.5 24718 latest/stable canonical** snapd
snapd-desktop-integration 0.9 253 latest/stable/… canonical** -
sublime-merge 2102 95 latest/stable snapcrafters* classic
thonny 4.1.7 239 latest/stable sameersharma2006 -
thunderbird 128.11.1esr-1 737 latest/stable canonical** -
todoist 9.17.0 1340 latest/stable doist** -
typescript-language-server 4.3.4 211 latest/stable alexmurray* -
yazi shipped 293 latest/stable sxyazi classic
yq v4.44.5 2634 latest/stable mikefarah -
zellij 0.42.2 41 latest/stable dominz88 classic

Installed Nix packages

❯ nix profile list | grep -Po "Name:[ ]*\K.+\$"
atuin
bash-language-server
cargo-udeps
deadnix
fzf
gh
gofumpt
nil
nixd
nixfmt-rfc-style
prettier
pyright
python-lsp-server
spread
starship
statix
taplo
terraform-ls
typos-lsp
vscode-langservers-extracted
yaml-language-server

I continue to find Nix development shells useful, and even use one to develop this site.

Configuration Management #

In the past I’ve used all kinds of scripts, Ansible playbooks and dotfile managers to solve this problem, and its a problem that was solved very elegantly by NixOS/Home Manager. I experimented with Home Manager on Ubuntu to manage dotfiles and configuration, but found the experience had more rough edges than I would like, and didn’t really adhere to my JFUI principles.

I’ve settled on a directory full of idempotent, single-purpose scripts which get executed by a wrapper named provision. This is somewhat inspired by Omakub, but without the menus and configuration they supply to allow their users some customisation. The scripts install packages, write configuration with tools like gsettings and symlink configuration into place where needed:

❯ ls
app-1password dev-python tool-agendrr
app-chrome dev-rust tool-atuin
app-flameshot font-meslo tool-gearlever
app-ghostty hw-yubikey tool-gh
app-obsidian kara-audioengine tool-git
app-pinta kara-backup tool-helix
app-rambox kara-data-disk tool-junction
app-signal kara-hiring-automation tool-lxd
app-sublime-merge kara-hiring-reports tool-multipass
app-thunderbird provision tool-podman
app-todoist system-flatpak tool-spread
configs system-fs tool-starship
dev-charms system-gnome tool-syncthing
dev-containers system-gnome-extensions tool-tailscale
dev-go system-nix tool-zed
dev-node system-shell tool-zellij

An example of a specific script, such as tool-zellij:

1
2
3
4
5
6


#!/usr/bin/env bash
set -e
DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
sudo snap install --classic zellij
mkdir -p "${HOME}/.config/zellij"
ln -sf "${DIR}/configs/zellij/config.kdl" "${HOME}/.config/zellij/config.kdl"

The provision script is similarly simple (and naive):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


#!/usr/bin/env bash
set -ex
DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )

sudo apt-get update
sudo apt-get upgrade -y

categories="system hw dev tool app font $(hostname)"
for c in $categories; do
 for x in ${c}-*; do
 source $x
 done
done

This is just about satisfactory. I’ve got into the habit of only ever installing software by creating the relevant script and configuration. If I don’t need it to persist, the work gets done in an ephemeral LXD container/VM then thrown away.

I’m working on a better solution for this named miso, short for “Make It So”. This a homegrown, multi-host configuration management tool for my machines that draws inspiration from cloud-init, home-manager, terraform and a few others. I’ll write about that in a future post when the code is a little more complete, but there is an example of an early configuration format I’m targeting available as a gist.

Everything in the linked config file is currently implemented and working with a decent suite of integration tests - but I’ve got a lot of tidying to do with error handling and such before I release it.

So What About Nix? #

I thoroughly enjoyed my adventures with Nix, and I consider having learned how to package software with Nix and use the available tooling to manage servers, desktops and development shells to have been incredibly worthwhile.

Even if I were to never touch Nix again, the general packaging and distribution engineering skills I learned have been invaluable, and I’m grateful to everyone who helped me on that journey through Matrix chats, Pull Requests and Mastodon interactions.

I remain active on the ~35 packages in nixpkgs for which I’m the maintainer. I continue to use Nix for development shells, CI and for certain packages on my Ubuntu machines. I have archived my flake for now because it’s not being maintained, but I’ve left it there in case there are any patterns that might be useful for others.

Summary #

Ubuntu has been very stable on my desktop, server and both of my laptops. I’ve enjoyed the level of integration and polish that comes with no effort in the desktop environment, and managing less configuration has been a freeing experience - even if some of my apps no longer have matching themes 😱.

Provisioning and configuration management are less structured and more cumbersome in Ubuntu, but that has driven me to build my own tool which was good fun, and gave me a nice challenge to solve through my journey learning Rust! I hope to learn from the project in a way that helps inform the development of Ubuntu itself in future releases.

If you’re into numbers, here are the stats for my renewed “config” directory, which contains all of the text-based configuration & scripts for my machines:

----------------------------------------------------------
Language files blank comment code
----------------------------------------------------------
Bourne Again Shell 47 154 81 413
TOML 3 13 1 53
Fish Shell 1 13 3 38
JSON 1 0 0 35
----------------------------------------------------------
SUM: 52 180 85 539
----------------------------------------------------------

From 3982 lines, to 539 lines, and much of that could be reduced if it wasn’t for the slightly repetitive nature of maintaining separate, idempotent scripts. Not bad.

As much as I feared the transition, my journey back to Ubuntu has been very enjoyable. I’m not “quitting Nix” or “over it”, but at least for now I’m enjoying a less complex existence with my personal computers.

Thanks for reading!

Packaging the Multipass Flutter GUI for NixOS

Thu, 16 Jan 2025 00:00:00 +0000

Introduction #

The very first application I ever packaged in nixpkgs was Multipass. Multipass is, according to the website:

a CLI to launch and manage VMs on Windows, Mac and Linux that simulates a cloud environment with support for cloud-init.

I wrote in some detail about how I use Multipass and LXD on my workstation to create and manage VMs for testing and development. Last year Multipass shipped a new GUI written from the ground up in Flutter. It provides a clean, modern way to launch, manage and interact with VMs. When the GUI first shipped, I briefly attempted to get it to build with Nix, but some combination of my lack of knowledge, and the maturity of the Flutter tooling in nixpkgs at the time meant I never finished it.

As version 1.15.0 was in its release candidate stage, I decided to have another go!

This post will break down the process into steps, but if you’d like the tl;dr, take a look at the pull request.

Housekeeping #

Before getting started on packaging the GUI, I did some housekeeping on the Multipass package. First order of business was to simply bump the version to 1.15.0 and ensure the package still built, which it did.

Last year, RFC 140 was introduced to simplify the directory structure of nixpkgs, introducing a new pkgs/by-name directory which will (eventually) render pkgs/top-level/all-packages.nix useless. This new structure will make finding package definitions much easier, as they’ll all be arranged alphabetically by the first two characters of their name. Using the Multipass example, rather than pkgs/tools/virtualization/multipass/default.nix, the new preferred path would be pkgs/by-name/mu/multipass/package.nix. There was a talk at NixCon 2023 which summarised this nicely if you’d like more information.

As I was going to be carrying out some hefty work on Multipass by introducing the GUI, I took the opportunity to move to the new scheme as part of the changes.

Restructuring #

The Multipass package carries a few patches, and is already reasonably involved, so I decided that the best route forward was to create separate files for multipassd/multipass (the server & command line client binaries, written in C++, and already packaged) and multipass.gui (the new Flutter application). Thus the new structure for the package looks like so:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


.
# Derivation files
├── gui.nix
├── multipassd.nix
├── package.nix
# Lock file for Fluter app
├── pubspec.lock.json
# Update script for daemon/cli & GUI
├── update.sh
# Patches
├── cmake_no_fetch.patch
├── cmake_warning.patch
├── lxd_socket_path.patch
├── test_unreachable_call.patch
└── vcpkg_no_install.patch

Using `symlinkJoin` #

This package now takes a more unusual shape. Because the GUI is essentially a entirely separate application, written with a different toolkit in a different language, but I wanted the two to be installed as part of the overall multipass package, I decided to use two completely separate derivations and use symlinkJoin to bundle the two together. According to the manual, symlinkJoin:

can be used to put many derivations into the same directory structure. It works by creating a new derivation and adding symlinks to each of the paths listed. It expects two arguments, name, and paths. name (or alternatively pname and version) is the name used in the Nix store path for the created derivation. paths is a list of paths that will be symlinked. These paths can be to Nix store derivations or any other subdirectory contained within.

This is particularly useful for the Multipass package, because the Multipass GUI dynamically-links against libraries that are built as part of the multipassd derivation, and linking the two derivations in this way ensures that all those files are located correctly at runtime without any extra plumbing. This allowed me to keep the top-level package.nix relatively simple (annotated below):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55


{
# ... snip ...
}:
let
 name = "multipass";
 version = "1.15.0";

 # Fetch the source just once, to be used in both the multipassd and GUI derivations
 multipass_src = fetchFromGitHub {
 owner = "canonical";
 repo = "multipass";
 rev = "refs/tags/v${version}";
 hash = "sha256-RoOCh1winDs7BZwyduZziHj6EMe3sGMYonkA757UrIU=";
 fetchSubmodules = true;
 };

 # Shared metadata between derivations.
 commonMeta = {
 homepage = "https://multipass.run";
 license = lib.licenses.gpl3Plus;
 maintainers = with lib.maintainers; [ jnsgruk ];
 platforms = [ "x86_64-linux" ];
 };

 # Build the multipass server & client binaries from the shared source
 # and common metadata.
 multipassd = callPackage ./multipassd.nix {
 inherit commonMeta multipass_src version;
 };

 # Build the multipass GUI using shared source and common metadata.
 multipass-gui = callPackage ./gui.nix {
 inherit commonMeta multipass_src multipassd version;
 };
in
# Create the top-level `multipass` package using `symlinkJoin`
symlinkJoin {
 inherit version;
 pname = name;

 # Join both of the newly created derivations
 paths = [ multipassd multipass-gui ];

 # Ensure tests are run, and define a (new) update script.
 passthru = {
 tests = lib.optionalAttrs stdenv.hostPlatform.isLinux {
 inherit (nixosTests) multipass;
 };
 updateScript = ./update.sh;
 };

 meta = commonMeta // {
 description = "Ubuntu VMs on demand for any workstation";
 };
}

Building the GUI #

Normally the server, client and GUI binaries are all built as part of a single invocation of cmake. We can’t do that here, so the early code I put into the multipass package when I failed to package the GUI before still stood - essentially stopping cmake from trying to build the Flutter GUI:

1
2
3
4


{
 # We'll build the flutter application separately using buildFlutterApplication
 cmakeFlags = [ "-DMULTIPASS_ENABLE_FLUTTER_GUI=false" ];
}

Next, I created gui.nix which uses the pkgs.buildFlutterApplication helper. The derivation is annotated below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67


{
 # Passed in from the package.nix shown above
 commonMeta,
 multipass_src,
 multipassd,
 version,
 # ... snip ...
}:
flutter327.buildFlutterApplication {
 inherit version;
 pname = "multipass-gui";
 src = multipass_src;

 # Ensure we build in the correct directory for the GUI code.
 sourceRoot = "${multipass_src.name}/src/client/gui";

 # To make the builds as deterministic as possible, nixpkgs reads the pubspec.lock
 # file for the Flutter application and downloads/caches the correct versions of
 # flutter packages to be used in the build. It does this with `pub2nix`.
 #
 # Here I use a JSON representation of the pubspec.lock file committed into nixpkgs.
 pubspecLock = lib.importJSON ./pubspec.lock.json;

 # Some of the Flutter dependencies used are pulled from Github (branches) directly, so this
 # ensures that the versions/commits are pinned for future builds.
 gitHashes = {
 dartssh2 = "sha256-2pypKwurziwGLZYuGaxlS2lzN3UvJp3bRTvvYYxEqRI=";
 hotkey_manager_linux = "sha256-aO0h94YZvgV/ggVupNw8GjyZsnXrq3qTHRDtuhNv3oI=";
 system_info2 = "sha256-fly7E2vG+bQ/+QGzXk+DYba73RZccltdW2LpZGDKX60=";
 tray_menu = "sha256-riiAiBEms+9ARog8i+MR1fto1Yqx+gwbBWyNbNq6VTM=";
 window_size = "sha256-71PqQzf+qY23hTJvcm0Oye8tng3Asr42E2vfF1nBmVA=";
 xterm = "sha256-h8vIonTPUVnNqZPk/A4ZV7EYCMyM0rrErL9ZOMe4ZBE=";
 };

 # ... snip ...

 # The Multipass GUI relies on protobuf for API communication; ensure the
 # Dart protobuf definitions are compiled before we build.
 preBuild = ''
 mkdir -p lib/generated

 # Generate the Dart gRPC code for the Multipass GUI.
 protoc \
 --plugin=protoc-gen-dart=${lib.getExe protoc-gen-dart} \
 --dart_out=grpc:lib/generated \
 -I ../../rpc \
 ../../rpc/multipass.proto \
 google/protobuf/timestamp.proto
 '';

 # Libraries built by multipassd are linked dynamically at runtime; multipassd must
 # be installed for the GUI to function.
 runtimeDependencies = [ multipassd ];

 # Housekeeping to ensure the icons/desktop files are correctly installed.
 postFixup = ''
 mv $out/bin/multipass_gui $out/bin/multipass.gui

 install -Dm444 $out/app/multipass-gui/data/flutter_assets/assets/icon.png \
 $out/share/icons/hicolor/256x256/apps/multipass.gui.png

 cp $out/share/applications/multipass.gui.autostart.desktop \
 $out/share/applications/multipass.gui.desktop
 '';

 # ... snip ..
}

Update Script #

That was enough to ensure that the GUI worked correctly. My concern now lied with the maintenance of the package. The process for updating the derivation now involved:

Bumping the version string
Updating the source hashes
Updating the pubspec.lock.json file
Updating the gitHashes of the Flutter dependencies

This seemed a little convoluted and error prone to handle manually each time, so I decided to implement an update script for the package to take care of future version bumps. Thankfully nixpkgs has good provision for this, allowing package maintainers to specify passthru.updateScript to a derivation, which I did here.

I won’t post the full update script here, as it’s lots of unsightly bash! Essentially the script takes care of determining the latest version released on Github by using curl, then updates the pubspec.lock.json using a combination of curl/yj, then finally works through updating the source hash for the Multipass repo, gRPC fork it uses, and each of the git hashes for the Flutter dependencies.

This can be invoked from within the nixpkgs repo with:

1

nix-shell maintainers/scripts/update.nix --argstr package multipass

Summary #

Overall, this turned out cleaner than I expected, though it took me longer to figure out than this post might suggest. My hope is that this will a useful reference to anybody trying to package Flutter applications with Nix, and if you’re a user of the multipass package, check out the shiny new GUI that’s available to you! The GUI is available from 1.15.0 onwards, which at the time of writing means you’ll need to install the package from nixos-unstable or similar (sadly I didn’t land the change in time for NixOS 24.11).

In my opinion, the Multipass team have done a lovely job on the GUI - and for many users this will provide a much improved onboarding experience. Nice work 😎

Experimenting with Rust, Nix, K6 and Parca

Sun, 01 Dec 2024 00:00:00 +0000

Introduction #

Over the past couple of weeks, I’ve been teaching myself Rust. I don’t have a pressing need to write much Rust right now, but I’m intrigued by the promises of memory safety, and have been increasingly impressed at the quality of some of the software that the community produces. I also think that the concepts popularised by Rust, such as the borrow checker, will stick around in computing for many years to come and I’d like to have more hands-on experience with that.

The Rust language also encourages the ideas of safety and soundness - sound code is (approximately) code that can’t cause memory corruption or exhibit undefined behaviour. You can read more in this excellent post Safety and Soundness in Rust.

This blog post started out life as a post about Rust and my experience learning it, but I got interested in the performance of the server implementation I came up with and the post evolved into a post more about profiling and load testing!

Learning Rust #

It’s been a little while since I dug into a new programming language, so I thought I’d mention how I went about it. In general I’m someone who learns best “by doing”. I normally try to read through some of the basic concepts, then jump to a project that will enable me to exercise them quite quickly. Fortunately, Rust has an excellent official guide in the form of the Rust Book, which covers everything from the obligatory Hello, World!, to concurrency, memory safety, publishing packages on crates.io and more. I thoroughly recommend it.

I’ve also picked up a copy of Rust for Rustaceans which was recommended by a couple of different colleagues - I intend to work through this next.

So, after working through the Rust Book over the course of about a week in my spare time, I needed a project!

Rewriting `gosherve` in Rust #

In my first post on this blog I talked about a small Go project I wrote several years ago named gosherve. This was one of my first Go projects - a simple web server that can serve some static assets, and a set of short-links/redirects which are specified in a Github Gist. It’s been happily running my website for several years, and it felt like a small, but ambitious enough project for my first adventure into Rust - particuarly as over the years gosherve has grown Prometheus metrics and the ability to serve assets from an embedded filesystem.

As I anticipated, naming the new project was the hardest part. I landed on servy, at least for now. I’m reasonably happy with the code - at the time of writing:

It can serve redirects
It can serve embedded web assets
It provides a metrics server on a separate port
It has reasonable unit/integration test coverage

One of the things I love about Go is the built-in ability to embed static assets into a binary through the //go:embed directive, which gives you a pointer to an embedded filesystem. I was able to achieve a similar effect in Rust with axum-embed, which in turn builds upon rust-embed.

I used Nix to create different variants of the build (i.e. a “vanilla” build, and one that serves my website by creating a derivation just for the web content, and another for the jnsgruk binary).

You can run a copy of this website (frozen in time just before I wrote this post!) by running nix run github:jnsgruk/servy#jnsgruk, or build a container for it by running nix build github:jnsgruk/servy#jnsgruk-container.

I was expecting the Rust binary to be quite a lot smaller. I’m not sure why. The old (Go) binary weighs in at 68MB, where the new Rust binary comes in at 67MB.

So I was right! I guess?! The result here is relatively uninteresting - a good chunk of both binaries is just the static assets (images!) that make up this site. At the time of writing the jnsgruk-content derivation evaluates at around 57MB - meaning there is 10MB and 9MB respectively for Go and Rust added by the actual server code.

Basic Load Testing with `k6` #

I’ve been looking for an excuse to play with k6 for a while. According to the documentation:

Grafana k6 is an open-source, developer-friendly, and extensible load testing tool. k6 allows you to prevent performance issues and proactively improve reliability.

In very simple terms, to use k6 you define a script (in Javascript) that outlines a set of requests to make, their success criteria and (optionally) the strategy for ramping up load on the server. It has many more features than I used for this project, but I was impressed with how simple it was to get started. I began by running the following (grabbing k6 from nixpkgs):

1
2
3
4
5


# Start a shell with k6 available
❯ nix shell github:NixOS/nixpkgs/nixos-unstable#k6
# Init a new k6 script
❯ k6 new
Initialized a new k6 test script in script.js. You can now execute it by running `k6 run script.js`.

Now I just needed to work out what I wanted to test! I wanted a relatively simple test that requested a mix of web assets and redirects for a sustained period, to see how much throughput I could achieve with each of the server implementations. The template came with some sensible setup for the number of VUs (virtual users) and duration of the test:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


import { check } from "k6"
import http from "k6/http"

export const options = {
 // A number specifying the number of VUs to run concurrently.
 vus: 10,
 // A string specifying the total duration of the test run.
 duration: "30s",
}
// ...

VUs are an emulation of a user interacting with your service; each of them is an agent which will execute the test script. Each time the script is executed (by a VU), that’s known as an “iteration”.

I wanted to test a variety of requests, so I first generated a list of valid URLs for my server. I used cariddi which is a web crawler written in Go, combined with jq to create a list of valid paths in a JSON file:

1
2
3
4
5
6
7


# Start a shell with k6 available
❯ nix shell github:NixOS/nixpkgs/nixos-unstable#cariddi
# Run cariddi and generate the paths.json
❯ echo http://localhost:8080 \
 | cariddi -- -plain \
 | cut -d"/" -f4- \
 | jq -r -nR '[inputs | select(length>0)]' > paths.json

Similarly I generated a list of valid redirects:

1
2
3


❯ curl -s "https://gist.githubusercontent.com/jnsgruk/b590f114af1b041eeeab3e7f6e9851b7/raw" \
 | cut -d" " -f1 \
 | jq -r -nR '[inputs | select(length>0)]' > redirects.json

Now in my k6 script, I was able to read those files to create a list of URLs to batch GET during the load test:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


// Allow the host to be overridden by the 'K6_HOST' env var.
const host = __ENV.K6_HOST || "http://localhost:8080"
// Read the list of paths/redirects.
const assets = JSON.parse(open(`./paths.json`))
const redirects = JSON.parse(open(`./redirects.json`))

export default function () {
 // Batch requests to redirects and static assets.
 const responses = http.batch(
 [...assets, ...redirects].map((p) => {
 return ["GET", `${host}/${p}`, null, { redirects: 0 }]
 })
 )

 // For each response, ensure we get a 200/301/308/404
 // response - we shouldn't see anything else.
 responses.forEach((r) => {
 check(r, {
 status: (res) => {
 return [200, 301, 308, 404].includes(res.status)
 },
 })
 })
}

I’m using a pretty naive check function here, though they can be expanded to check for other conditions in the response body, the size of the response, etc.

You can see my final test script on Github.

Finally, I ran k6 and let it rip! This initial run was against servy, running on my workstation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40


❯ k6 run script.js

 /\  Grafana /‾‾/
 /\  / \  |\  __ / /
 / \/ \  | |/ / / ‾‾\
 / \  | ( | (‾) |
 / __________ \  |_|\_\  \_____/

 execution: local
 script: script.js
 output: -

 scenarios: (100.00%) 1 scenario, 10 max VUs, 1m0s max duration (incl. graceful stop):
 * default: 10 looping VUs for 30s (gracefulStop: 30s)

INFO[0030] [k6-reporter v2.3.0] Generating HTML summary report source=console

 ✓ status

 checks.........................: 100.00% ✓ 1331889 ✗ 0
 data_received..................: 148 GB 4.9 GB/s
 data_sent......................: 163 MB 5.4 MB/s
 http_req_blocked...............: avg=2.41µs min=390ns med=1.72µs max=17.3ms p(90)=3.6µs p(95)=4.91µs
 http_req_connecting............: avg=4ns min=0s med=0s max=462.21µs p(90)=0s p(95)=0s
 http_req_duration..............: avg=501.25µs min=42.08µs med=139.77µs max=45.41ms p(90)=291.59µs p(95)=419.65µs
 { expected_response:true }...: avg=501.25µs min=42.08µs med=139.77µs max=45.41ms p(90)=291.59µs p(95)=419.65µs
 http_req_failed................: 0.00% ✓ 0 ✗ 1331889
 http_req_receiving.............: avg=379.39µs min=4.13µs med=29.07µs max=45.23ms p(90)=108.31µs p(95)=177.05µs
 http_req_sending...............: avg=4.45µs min=960ns med=3.48µs max=3.36ms p(90)=6.83µs p(95)=8.79µs
 http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
 http_req_waiting...............: avg=117.4µs min=33.65µs med=98.19µs max=23.03ms p(90)=176.34µs p(95)=225.09µs
 http_reqs......................: 1331889 44333.219428/s
 iteration_duration.............: avg=38.08ms min=4.92ms med=45.46ms max=80.35ms p(90)=47.68ms p(95)=48.53ms
 iterations.....................: 7881 262.326742/s
 vus............................: 10 min=10 max=10
 vus_max........................: 10 min=10 max=10


running (0m30.0s), 00/10 VUs, 7881 complete and 0 interrupted iterations
default ✓ [======================================] 10 VUs 30s

Nice! Between the 10 VUs we configured, k6 managed nearly 8000 iterations, and my new server responded with a total of 148GB of data!

Nix-ified Load Testing #

Now I’d figured out the basics of k6, I wanted to create some infrastructure that would allow me to run load tests in consistent environments against both old and new implementations of my server.

A couple of ideas came to mind here - the first of which was the NixOS integration test driver. I absolutely love this feature of Nix, and the scripts for interacting with the driver are nice and simple to maintain. One slight irritation in this case is that the test machines don’t have access to the internet - which is where the redirects map is fetched from in my precompiled server binaries. It’s certainly possible to get clever with a fake redirects server and some DNS shennanigans in the test machines, but I opted instead to build simple nixosConfigurations which could be started as VMs, similar to the approach taken in a previous post.

I wanted to automate:

The creation of a VM with both the Go and Rust versions of my server
The generation of the paths.json and redirects.json files I created above
Running the k6 load test
Fetching the results from the load test

Defining Test VMs #

I created a new repository with a flake.nix which took nixpkgs, my jnsgr.uk repo and my servy repo as inputs, pinned to the latest revisions at the time of writing:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


{
 description = "web server benchmarking flake";

 inputs = {
 nixpkgs.url = "github:nixos/nixpkgs/nixpkgs-unstable";
 jnsgruk-go.url = "github:jnsgruk/jnsgr.uk/3240a0104c01ae672a6f5f7b0529ad08bcbc8af2";
 jnsgruk-rust.url = "github:jnsgruk/servy/71a337317defc779c6c55d486e20d104c5d478f2";
 };

 # ...
}

Next up I needed a virtual machine definition, which I first defined as a nixosConfiguration in the flake’s outputs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


{
 # ... inputs ...
 outputs = { self, nixpkgs, jnsgruk-go, jnsgruk-rust, ... }:
 let
 forAllSystems = nixpkgs.lib.genAttrs [ "x86_64-linux" "aarch64-linux" ];
 in
 {
 # A minimal NixOS virtual machine which used for testing craft applications.
 nixosConfigurations = forAllSystems (system: {
 benchvm = nixpkgs.lib.nixosSystem {
 specialArgs = {
 inherit self system;
 jnsgruk-go = jnsgruk-go.packages.${system}.jnsgruk;
 jnsgruk-rust = jnsgruk-rust.packages.${system}.jnsgruk;
 };
 modules = [ ./vm.nix ];
 };
 });
 };
}

The actual machine configuration lives in vm.nix. This is a standard NixOS configuration, with the addition of some elements that define the specs of the virtual machine used to boot it (cores, memory, disk, etc.). In this case the machine is configured with just a root user, and the password password. This is not an ideal setup from a security standpoint, but these VMs were only run on my workstation (behind NAT) for a short period of time and the plain text password eased the automation I’m about to describe. A more robust approach would have been to put my SSH public keys into the authorized keys definition for the user, but then you folks wouldn’t have been able to play along as easily!

In this particular configuration, the VM is set up like so:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


virtualisation = {
 forwardPorts = [{
 from = "host";
 host.port = 2222;
 guest.port = 22;
 }];
 cores = 4;
 memorySize = 4096;
 diskSize = 10000;
};

With that in place, we can now boot the VM (in the background):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


❯ nix run .#nixosConfigurations.benchvm.config.system.build.vm -- --daemonize --display none
❯ ssh -p 2222 root@localhost
The authenticity of host '[localhost]:2222 ([127.0.0.1]:2222)' can't be established.
ED25519 key fingerprint is SHA256:icnH3EQAzmjdfCkyPWFljQWaVSCaXdP2M+ekKXd0NlY.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[localhost]:2222' (ED25519) to the list of known hosts.
(root@localhost) Password:

[root@benchvm:~]#

Wrapping `k6` #

Now on to automating the test itself. When configuring the VM, I made sure to include a systemd unit for each of the server implementations, so next I wanted to write a simple script that would execute the load test for each implementation and fetch the results.

Before that, I modified the k6 script to not only output a report to stdout, but also to a text file, a JSON file and a rendered HTML file (I didn’t know at the time which I’d prefer, and I wanted to survey the options!). I achieved this by defining the handleSummary function in the k6 script:

1
2
3
4
5
6
7
8


export function handleSummary(data) {
 return {
 "summary.json": JSON.stringify(data),
 "summary.html": htmlReport(data),
 "summary.txt": textSummary(data, { enableColors: false, indent: "" }),
 stdout: "\n" + textSummary(data, { enableColors: true, indent: "" }) + "\n\n",
 }
}

First I scripted the generation of the paths and the execution of the test script, and packaged the combination up as a Nix package for installation into the test machine. This is just a bash script which ensures only the specified implementation is running (with systemd), generates the list of URLs to test and then runs k6 against the server. That script is packaged as a Nix package called k6test which contains both my bash script and the k6 test script; the Nix package is then added to the VM configuration.

Automating Test Execution #

I borrowed the vm-exec script from a past project, renaming it benchvm-exec, and created a new script called benchvm-test:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


#!/usr/bin/env bash
set -euo pipefail
info() { echo -e "\e[92m[$HOSTNAME] $*\e[0m"; }

# Set some SSH options to ignore host key errors and make logging quieter.
# This is a bad idea in general, but here is used to faciliate comms with
# a brand new VM each time.
SSH_OPTS=(-P 2222 -o "UserKnownHostsFile=/dev/null" -o "StrictHostKeyChecking=no")

info "Running k6test against $1"
benchvm-exec k6test "$1"

info "Collecting results files"
sshpass -ppassword scp "${SSH_OPTS[@]}" root@localhost:summary.json "summary-${1}.json"
sshpass -ppassword scp "${SSH_OPTS[@]}" root@localhost:summary.html "summary-${1}.html"
sshpass -ppassword scp "${SSH_OPTS[@]}" root@localhost:summary.txt "summary-${1}.txt"

info "Results available in summary-${1}.txt, summary-${1}.json and summary-${1}.html"

This was the final piece of the puzzle for now; after building and packaging this script and defining the test VM as a package in the flake, I’d enabled the following workflow:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Start the devShell for the server-bench project
❯ nix develop github:jnsgruk/server-bench
# Build & run the VM
❯ run-benchvm-vm --daemonize --display none
# Start, and load test the gosherve based server
❯ benchvm-test jnsgruk-go
# Start, and load test the Rust based server
❯ benchvm-test jnsgruk-rust
# All done, power down the VM
❯ benchvm-exec poweroff

Excellent! Now on to some actual testing!

Initial Load Test Results #

Having seen the results for servy earlier, I started up the existing gosherve based server and ran the same test, only to discover quite a delta in the results. Where servy managed 4.9 GB/s throughout the test, gosherve only achieved 697 MB/s (sending 21GB in total). The full results are in the details box below, but overall it managed much lower performance across the board on the measurements that k6 makes.

Initial gosherve load-test results

checks.........................: 100.00% ✓ 189449 ✗ 0
data_received..................: 21 GB 697 MB/s
data_sent......................: 26 MB 848 kB/s
http_req_blocked...............: avg=2.02µs min=400ns med=1.42µs max=19.76ms p(90)=2.29µs p(95)=2.77µs
http_req_connecting............: avg=203ns min=0s med=0s max=19.52ms p(90)=0s p(95)=0s
http_req_duration..............: avg=9.3ms min=29.49µs med=9.36ms max=152.39ms p(90)=13.16ms p(95)=14.76ms
{ expected_response:true }...: avg=9.3ms min=29.49µs med=9.36ms max=152.39ms p(90)=13.16ms p(95)=14.76ms
http_req_failed................: 0.00% ✓ 0 ✗ 189449
http_req_receiving.............: avg=106.56µs min=4.88µs med=48.7µs max=6.01ms p(90)=215.06µs p(95)=381.38µs
http_req_sending...............: avg=4.81µs min=950ns med=3.97µs max=19.71ms p(90)=5.93µs p(95)=6.98µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=9.19ms min=22.85µs med=9.26ms max=152.2ms p(90)=12.96ms p(95)=14.51ms
http_reqs......................: 189449 6281.256789/s
iteration_duration.............: avg=268.14ms min=176.88ms med=250.61ms max=433.23ms p(90)=334.14ms p(95)=342.1ms
iterations.....................: 1121 37.1672/s
vus............................: 10 min=10 max=10
vus_max........................: 10 min=10 max=10

I was quite surprised by this: my expectation was that the Go HTTP server would outperform what I’d put together in Rust. I decided to look a little deeper and see if I could figure out why, or at least why the delta was so big.

Profiling `gosherve` with Parca #

For last couple of years, I’ve been playing with Parca, which is a continuous profiling tool written in Go by the folks at Polar Signals. I wrote about it back in 2022 on Charmhub, demonstrating how Parca could be used for profiling applications deployed with Juju.

Instrumenting `gosherve` #

There are two components to the open source Parca offering - the server backend, and a zero-instrumentation, eBPF-based agent which can be used for on-CPU profiling of any workload. Because gosherve is Go based, I didn’t need the agent so long as I enabled the pprof endpoint in my server, which is trivial for almost any Go application. I took main.go and applied the following changes to ensure that I could hit the pprof endpoints on port 6060:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


diff --git a/main.go b/main.go
index 964ceb2..374c440 100644
--- a/main.go
+++ b/main.go
@@ -10,6 +10,9 @@ import (
 "log/slog"
 "os"

+ "net/http"
+ _ "net/http/pprof"
+
 "github.com/jnsgruk/gosherve/pkg/logging"
 "github.com/jnsgruk/gosherve/pkg/server"
 )
@@ -29,6 +32,11 @@ func main() {
 flag.Parse()
 logging.SetupLogger(*logLevel)

+ go func() {
+ http.ListenAndServe(":6060", nil)
+ slog.Info("pprof server started on port 6060")
+ }()
+
 // Create an fs.FS from the embedded filesystem
 fsys, err := fs.Sub(publicFS, "public")
 if err != nil {

Next I fired up Parca (having first packaged it for NixOS 😉) and configured it to scrape the new jnsgruk binary while I ran the same test against it, which resulted in the following profile (which you can explore yourself on pprof.me):

Looking at this, we can see that around a third of the time was spent in garbage collection. This seemed high; it was more prominent on this run than others but it was persistently a high portion of the CPU time. Something to loop back to! However, inside the route handler itself there are two things that stood out to me:

Initial optimisations #

In the red box, you can see about a third of the time spent in the route handler was in creating SHA1 sums, which is happening here to calculate the ETag for content before serving it. This could probably be optimised - either by changing how the ETag is calculated (i.e. not hashing the whole file contents), or by caching ETags in memory, particularly given that in the case of my website, gosherve only serves assets which are embedded (read only) in the binary, so the ETag will never change for a given file for a particular compiled binary.

In the blue box, we can see that another third of the time spent is on refreshing the redirects. This seemed strange; the redirects don’t change throughout the life of the test, and we’re only requesting known redirects from our redirects.json file. This seemed much more likely to be a culprit in the poor load test results, because unlike calculating a SHA1 sum (which is almost entirely CPU-bound), this function is making an outbound (TLS) connection to Github and potentially blocking on IO before parsing the response, rechecking the redirects map for a match, etc.

As I looked at the code, I noticed an obvious (and somewhat embarrassing!) mistake I’d made when implementing gosherve. Because the server tries to parse redirects before looking for files with a matching path, every request for a file causes the redirects map to be updated from its upstream source 🤦. Looking back at the new servy implementation, I hadn’t made the same mistake - perhaps this was the reason the Rust version was so much faster?

I made a small change to gosherve and re-ran the test, which resulted in this profile and quite a substantial bump in request performance!

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


checks.........................: 100.00% ✓ 913783 ✗ 0
data_received..................: 101 GB 3.4 GB/s
data_sent......................: 123 MB 4.1 MB/s
http_req_blocked...............: avg=3.21µs min=390ns med=2.12µs max=14.12ms p(90)=4.19µs p(95)=5.7µs
http_req_connecting............: avg=70ns min=0s med=0s max=14.09ms p(90)=0s p(95)=0s
http_req_duration..............: avg=1.8ms min=42.12µs med=899.13µs max=31.31ms p(90)=4.86ms p(95)=6.45ms
 { expected_response:true }...: avg=1.8ms min=42.12µs med=899.13µs max=31.31ms p(90)=4.86ms p(95)=6.45ms
http_req_failed................: 0.00% ✓ 0 ✗ 913783
http_req_receiving.............: avg=261.88µs min=4.85µs med=63.59µs max=17.8ms p(90)=620.15µs p(95)=1.31ms
http_req_sending...............: avg=8.21µs min=1.05µs med=5.21µs max=8.73ms p(90)=9.87µs p(95)=13.07µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=1.53ms min=30.81µs med=656.71µs max=30.76ms p(90)=4.38ms p(95)=5.91ms
http_reqs......................: 913783 30435.133157/s
iteration_duration.............: avg=55.5ms min=28.86ms med=55.18ms max=77.07ms p(90)=61.02ms p(95)=62.98ms
iterations.....................: 5407 180.089545/s
vus............................: 10 min=10 max=10
vus_max........................: 10 min=10 max=10

The overall throughput increased by five times, with a decrease in latency across the board. Still around a third less overall throughput than the Rust server, but a huge improvement nonetheless.

I subsequently experimented with (naively) removing the ETag calculation to see whether or not it was worth implementing some caching - but it actually resulted in very little difference in throughput and CPU utilisation. Besides, while the calculation takes some CPU time, in a real-life deployment it reduces the overall load on the server by ensuring that http.ServeContent only sends actual content when there is a new version, relying more heavily on the user’s browser cache. My k6 tests (deliberately) weren’t sending the right headers (such as If-None-Match) to benefit from the behaviour one might expect from a browser, and it was just requesting the same files over and over again with the equivalent of a cold/empty cache.

Reducing Allocations #

Now back to all those allocations, and the time spent in garbage collection as a result… gosherve was spending about a third of it’s time in the io.ReadFile function, and given the amount of time spent in garbage collection, any optimisation made to the number of allocations on the heap would likely yield another big performance increase.

The problem lied in the routeHandler, where I was reading the entire contents of each file that was being served into memory. fs.ReadFile makes an allocation the size of the file it’s reading, meaning the contents of the file end up on the heap. In a situation like our load test - this means the entire contents of my website ended up on the heap, and the Go runtime was busily garbage collecting to clean up.

I looked around at alternative ways to implement the same functionality in a more efficient manner. In Go 1.22, http.ServeFileFS was introduced as a counterpart to http.ServeFile. http.ServeFileFS operates on an fs.FS rather than the host filesystem. Under the hood, rather than reading the whole file into memory ServeFileFS is using io.Copy and thus io.copyBuffer which allocates a single buffer that fits on the stack - fewer allocations means less memory usage, and less time spent occupying cores with garbage collection!

The change to gosherve was pretty simple - you can see a slightly shortened diff below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


diff --git a/pkg/server/route_handler.go b/pkg/server/route_handler.go
index 2417e45..e0c9d68 100644
--- a/pkg/server/route_handler.go
+++ b/pkg/server/route_handler.go
@@ -1,7 +1,6 @@
 // snip!

- // Try reading the file and return early if that fails
- b, err := fs.ReadFile(*s.webroot, filepath)
- if err != nil {
- return false
- }
-
 w.Header().Set("Cache-Control", "public, max-age=31536000, must-revalidate")
- w.Header().Set("ETag", fmt.Sprintf(`"%d-%x"`, len(b), sha1.Sum(b)))
+ w.Header().Set("ETag", fmt.Sprintf(`"%s-%d-%x"`, fi.Name(), fi.Size(), sha1.Sum([]byte(fi.ModTime().String()))))

- http.ServeContent(w, r, filepath, time.Now(), bytes.NewReader(b))
+ http.ServeFileFS(w, r, *s.webroot, filepath)
 s.metrics.responseStatus.WithLabelValues(strconv.Itoa(http.StatusOK)).Inc()
 l.Info("served file", slog.Group("response", "status_code", http.StatusOK, "file", filepath))

This change also necessitated a change to ETag calculation, which now only hashes the filename, size and modified time. There is a slight oddity in that files from an embedded filesystem have their timestamps all fixed (to avoid changes in static files messing with build reproducibility), but since the files cannot change during a binary’s lifetime, the ETag calculated from the modified timestamp is still just as valid.

I quickly made a new build of jnsgruk using a locally referenced copy of gosherve and ran k6/parca to check the outcome. You can explore the profile yourself on pprof.me:

Here you can clearly see that the runtime is spending much less time in garbage collection, and the call to io.Copy/io.copyBuffer without the calls to fs.ReadFile. It’s now also much harder to spot the ETag generation - likely because we’re now only hashing a small string, rather than the whole file contents.

The k6 results came back as follows 🚀:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


checks.........................: 100.00% ✓ 1525300 ✗ 0
data_received..................: 170 GB 5.7 GB/s
data_sent......................: 207 MB 6.9 MB/s
http_req_blocked...............: avg=4µs min=410ns med=2.36µs max=18.47ms p(90)=5.02µs p(95)=6.77µs
http_req_connecting............: avg=40ns min=0s med=0s max=17.98ms p(90)=0s p(95)=0s
http_req_duration..............: avg=999.6µs min=42.17µs med=645.03µs max=24.22ms p(90)=2.34ms p(95)=3.09ms
 { expected_response:true }...: avg=999.6µs min=42.17µs med=645.03µs max=24.22ms p(90)=2.34ms p(95)=3.09ms
http_req_failed................: 0.00% ✓ 0 ✗ 1525300
http_req_receiving.............: avg=245.43µs min=4.64µs med=75.29µs max=23.48ms p(90)=626.08µs p(95)=1.18ms
http_req_sending...............: avg=9.69µs min=1µs med=5.49µs max=19.54ms p(90)=10.86µs p(95)=14.46µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=744.47µs min=28.09µs med=395.81µs max=20.62ms p(90)=1.89ms p(95)=2.58ms
http_reqs......................: 1525300 50823.211595/s
iteration_duration.............: avg=34.41ms min=11.91ms med=34.44ms max=67.75ms p(90)=37.6ms p(95)=38.61ms
iterations.....................: 8716 290.418352/s
vus............................: 10 min=10 max=10
vus_max........................: 10 min=10 max=10

The Go server is now in the lead! There isn’t a huge margin here between the Rust server and the Go server, but this is about the sort of difference I was expecting based on the reading I’d done before doing the testing.

Profiling `servy`? #

I did profile servy using parca-agent, but I’ve yet to look at the results in detail. The profile is a lot more complex (see below), partially as a result of how much work is going on under the hood when standing up an axum server with hyper, tower and tokio. I’m going to spend some more time with this profile over the coming weeks, but you can explore it for yourself on pprof.me.

If you do play with the profile on pprof.me, try filtering by function name and entering servy, and you’ll see the parts of the code I wrote highlighted on the profile to explore:

By the time you read this, the parca-agent should be upstream into nixpkgs, but in the mean time you can pull the package I included in the server-bench repo and use that:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Start the devShell for the server-bench project
❯ nix develop github:jnsgruk/server-bench
# Start the agent
❯ sudo parca-agent -- \
 --remote-store-address="grpc.polarsignals.com:443" \
 --remote-store-bearer-token="$TOKEN" \
 --node="$HOSTNAME"

# Or, on Ubuntu (see polarsignals.com/docs/setup-collection-snaps)
❯ sudo snap install --classic parca-agent

In my Nix example above, I’m sending profiles to Polar Signals Cloud, but you can also set the remote-store-address to the address of a locally running parca instance no problem.

Depending on the language you’re profiling, you’ll either need to include debug info in the binary you’re profiling (which is what I did), or upload your source/debug info manually. There are more instructions for that in the docs.

Final Results & Comparisons #

I wanted to gather some test results from all three targets across a number of different machine configurations; my website runs on a teeny-tiny Fly.io instance with 1 vCPU and 256MB RAM, so I was interested to see how performance changed on machines with varying configurations. I’ve included a summary in the table below which highlights just the data_received measurement from each test, and you can see the full reports from k6 by expanding the details box:

Expand for full `k6` reports

1 vCPU/256MB RAM/1 VUs - gosherve (old)

checks.........................: 100.00% ✓ 23322 ✗ 0
data_received..................: 2.6 GB 86 MB/s
data_sent......................: 3.1 MB 105 kB/s
http_req_blocked...............: avg=2.41µs min=400ns med=1.28µs max=1.04ms p(90)=1.87µs p(95)=2.27µs
http_req_connecting............: avg=50ns min=0s med=0s max=508.13µs p(90)=0s p(95)=0s
http_req_duration..............: avg=7.57ms min=49.54µs med=7.71ms max=143.13ms p(90)=9.42ms p(95)=10.65ms
{ expected_response:true }...: avg=7.57ms min=49.54µs med=7.71ms max=143.13ms p(90)=9.42ms p(95)=10.65ms
http_req_failed................: 0.00% ✓ 0 ✗ 23322
http_req_receiving.............: avg=98.03µs min=4.46µs med=57.25µs max=7.69ms p(90)=177.19µs p(95)=250.6µs
http_req_sending...............: avg=5.7µs min=970ns med=3.1µs max=480.07µs p(90)=4.79µs p(95)=9.26µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=7.47ms min=37.26µs med=7.62ms max=142.9ms p(90)=9.24ms p(95)=10.33ms
http_reqs......................: 23322 776.881114/s
iteration_duration.............: avg=217.52ms min=187.06ms med=198.66ms max=359.52ms p(90)=276.89ms p(95)=280.19ms
iterations.....................: 138 4.59693/s
vus............................: 1 min=1 max=1
vus_max........................: 1 min=1 max=1

1 vCPU/256MB RAM/1 VUs - gosherve

checks.........................: 100.00% ✓ 158691 ✗ 0
data_received..................: 18 GB 587 MB/s
data_sent......................: 21 MB 714 kB/s
http_req_blocked...............: avg=1.89µs min=379ns med=1.08µs max=1.15ms p(90)=1.62µs p(95)=1.98µs
http_req_connecting............: avg=11ns min=0s med=0s max=655.05µs p(90)=0s p(95)=0s
http_req_duration..............: avg=1.04ms min=48.25µs med=537.57µs max=31.33ms p(90)=2.63ms p(95)=3.66ms
{ expected_response:true }...: avg=1.04ms min=48.25µs med=537.57µs max=31.33ms p(90)=2.63ms p(95)=3.66ms
http_req_failed................: 0.00% ✓ 0 ✗ 158691
http_req_receiving.............: avg=91.12µs min=3.66µs med=20.8µs max=23.05ms p(90)=156.29µs p(95)=287.17µs
http_req_sending...............: avg=6.47µs min=900ns med=2.57µs max=2.22ms p(90)=4.06µs p(95)=6.21µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=949.02µs min=35.88µs med=466.14µs max=31.3ms p(90)=2.46ms p(95)=3.47ms
http_reqs......................: 158691 5287.057691/s
iteration_duration.............: avg=31.95ms min=21.16ms med=30.39ms max=107.46ms p(90)=36.19ms p(95)=41.18ms
iterations.....................: 939 31.284365/s
vus............................: 1 min=1 max=1
vus_max........................: 1 min=1 max=1

1 vCPU/256MB RAM/1 VUs - servy

checks.........................: 100.00% ✓ 148213 ✗ 0
data_received..................: 16 GB 547 MB/s
data_sent......................: 18 MB 605 kB/s
http_req_blocked...............: avg=1.5µs min=350ns med=971ns max=1.11ms p(90)=1.48µs p(95)=1.82µs
http_req_connecting............: avg=11ns min=0s med=0s max=451.97µs p(90)=0s p(95)=0s
http_req_duration..............: avg=928.79µs min=69.84µs med=671.38µs max=55.59ms p(90)=1.57ms p(95)=2.18ms
{ expected_response:true }...: avg=928.79µs min=69.84µs med=671.38µs max=55.59ms p(90)=1.57ms p(95)=2.18ms
http_req_failed................: 0.00% ✓ 0 ✗ 148213
http_req_receiving.............: avg=136.98µs min=3.76µs med=17.14µs max=46.31ms p(90)=103.7µs p(95)=192.52µs
http_req_sending...............: avg=5.33µs min=890ns med=2.35µs max=2.99ms p(90)=3.69µs p(95)=5.05µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=786.47µs min=55.89µs med=629.47µs max=17.92ms p(90)=1.43ms p(95)=1.96ms
http_reqs......................: 148213 4940.080272/s
iteration_duration.............: avg=34.19ms min=16.57ms med=25.68ms max=109.45ms p(90)=55.08ms p(95)=57.83ms
iterations.....................: 877 29.231244/s
vus............................: 1 min=1 max=1
vus_max........................: 1 min=1 max=1

2 vCPU/4GB RAM/10 VUs - gosherve (old)

checks.........................: 100.00% ✓ 116272 ✗ 0
data_received..................: 13 GB 426 MB/s
data_sent......................: 16 MB 518 kB/s
http_req_blocked...............: avg=2.25µs min=380ns med=1.06µs max=4.3ms p(90)=1.69µs p(95)=2.05µs
http_req_connecting............: avg=475ns min=0s med=0s max=4.27ms p(90)=0s p(95)=0s
http_req_duration..............: avg=15.37ms min=29.8µs med=14.28ms max=145.82ms p(90)=24.79ms p(95)=29.78ms
{ expected_response:true }...: avg=15.37ms min=29.8µs med=14.28ms max=145.82ms p(90)=24.79ms p(95)=29.78ms
http_req_failed................: 0.00% ✓ 0 ✗ 116272
http_req_receiving.............: avg=174.41µs min=4.15µs med=28.84µs max=21.95ms p(90)=241.72µs p(95)=703.97µs
http_req_sending...............: avg=7.22µs min=890ns med=2.98µs max=5.58ms p(90)=5.8µs p(95)=9.15µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=15.19ms min=22.98µs med=14.11ms max=144.5ms p(90)=24.48ms p(95)=29.37ms
http_reqs......................: 116272 3837.442876/s
iteration_duration.............: avg=439.41ms min=323.27ms med=430.04ms max=567.74ms p(90)=505.87ms p(95)=516.96ms
iterations.....................: 688 22.706763/s
vus............................: 10 min=10 max=10
vus_max........................: 10 min=10 max=10

2 vCPU/4GB RAM/10 VUs - gosherve

checks.........................: 100.00% ✓ 362167 ✗ 0
data_received..................: 40 GB 1.3 GB/s
data_sent......................: 49 MB 1.6 MB/s
http_req_blocked...............: avg=2.23µs min=360ns med=770ns max=12.31ms p(90)=1.34µs p(95)=1.66µs
http_req_connecting............: avg=117ns min=0s med=0s max=1.84ms p(90)=0s p(95)=0s
http_req_duration..............: avg=4.7ms min=32.38µs med=3.94ms max=47.81ms p(90)=9.04ms p(95)=11.63ms
{ expected_response:true }...: avg=4.7ms min=32.38µs med=3.94ms max=47.81ms p(90)=9.04ms p(95)=11.63ms
http_req_failed................: 0.00% ✓ 0 ✗ 362167
http_req_receiving.............: avg=196.28µs min=3.38µs med=15.42µs max=25.62ms p(90)=142.41µs p(95)=896.21µs
http_req_sending...............: avg=8.44µs min=890ns med=2.19µs max=18.13ms p(90)=3.63µs p(95)=6.34µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=4.5ms min=25.57µs med=3.77ms max=45.46ms p(90)=8.74ms p(95)=11.13ms
http_reqs......................: 362167 12040.737924/s
iteration_duration.............: avg=140.21ms min=80.54ms med=139.82ms max=221.91ms p(90)=153.25ms p(95)=157.95ms
iterations.....................: 2143 71.24697/s
vus............................: 10 min=10 max=10
vus_max........................: 10 min=10 max=10

2 vCPU/4GB RAM/10 VUs - servy

checks.........................: 100.00% ✓ 464412 ✗ 0
data_received..................: 51 GB 1.7 GB/s
data_sent......................: 57 MB 1.9 MB/s
http_req_blocked...............: avg=1.99µs min=360ns med=720ns max=14.38ms p(90)=1.13µs p(95)=1.42µs
http_req_connecting............: avg=92ns min=0s med=0s max=6.71ms p(90)=0s p(95)=0s
http_req_duration..............: avg=3.67ms min=44.91µs med=3.07ms max=60.17ms p(90)=5.9ms p(95)=7.98ms
{ expected_response:true }...: avg=3.67ms min=44.91µs med=3.07ms max=60.17ms p(90)=5.9ms p(95)=7.98ms
http_req_failed................: 0.00% ✓ 0 ✗ 464412
http_req_receiving.............: avg=206.58µs min=3.41µs med=14.06µs max=56.96ms p(90)=56.97µs p(95)=143.43µs
http_req_sending...............: avg=7.12µs min=870ns med=1.62µs max=14.84ms p(90)=3.01µs p(95)=5.78µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=3.45ms min=35.61µs med=3.01ms max=24.64ms p(90)=5.7ms p(95)=7.54ms
http_reqs......................: 464412 15448.582291/s
iteration_duration.............: avg=109.3ms min=63.59ms med=108.88ms max=161.77ms p(90)=120.71ms p(95)=124.65ms
iterations.....................: 2748 91.41173/s
vus............................: 10 min=10 max=10
vus_max........................: 10 min=10 max=10

16 vCPU/16GB RAM/10 VUs - gosherve (old)

checks.........................: 100.00% ✓ 119990 ✗ 0
data_received..................: 13 GB 439 MB/s
data_sent......................: 16 MB 534 kB/s
http_req_blocked...............: avg=2.05µs min=440ns med=1.55µs max=972.62µs p(90)=2.32µs p(95)=2.74µs
http_req_connecting............: avg=38ns min=0s med=0s max=275.54µs p(90)=0s p(95)=0s
http_req_duration..............: avg=14.82ms min=35.49µs med=15.16ms max=148.26ms p(90)=21.12ms p(95)=23.89ms
{ expected_response:true }...: avg=14.82ms min=35.49µs med=15.16ms max=148.26ms p(90)=21.12ms p(95)=23.89ms
http_req_failed................: 0.00% ✓ 0 ✗ 119990
http_req_receiving.............: avg=104.7µs min=5.07µs med=43.71µs max=8.96ms p(90)=228.22µs p(95)=396.05µs
http_req_sending...............: avg=4.52µs min=940ns med=3.99µs max=944.81µs p(90)=6.28µs p(95)=7.86µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=14.71ms min=23.97µs med=15.05ms max=148.25ms p(90)=20.95ms p(95)=23.7ms
http_reqs......................: 119990 3957.203279/s
iteration_duration.............: avg=426.57ms min=343.33ms med=422.38ms max=526.97ms p(90)=491ms p(95)=502.47ms
iterations.....................: 710 23.415404/s
vus............................: 10 min=10 max=10
vus_max........................: 10 min=10 max=10

16 vCPU/16GB RAM/10 VUs - gosherve

checks.........................: 100.00% ✓ 1297244 ✗ 0
data_received..................: 144 GB 4.8 GB/s
data_sent......................: 175 MB 5.8 MB/s
http_req_blocked...............: avg=2.83µs min=419ns med=1.63µs max=7.97ms p(90)=2.95µs p(95)=3.78µs
http_req_connecting............: avg=2ns min=0s med=0s max=142.69µs p(90)=0s p(95)=0s
http_req_duration..............: avg=1.22ms min=31.96µs med=697.97µs max=36.9ms p(90)=3.16ms p(95)=4.07ms
{ expected_response:true }...: avg=1.22ms min=31.96µs med=697.97µs max=36.9ms p(90)=3.16ms p(95)=4.07ms
http_req_failed................: 0.00% ✓ 0 ✗ 1297244
http_req_receiving.............: avg=234.98µs min=4.21µs med=39.77µs max=35.6ms p(90)=640.71µs p(95)=1.24ms
http_req_sending...............: avg=7.19µs min=930ns med=3.91µs max=30.11ms p(90)=6.84µs p(95)=9.22µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=980.01µs min=19.74µs med=465.75µs max=16.64ms p(90)=2.75ms p(95)=3.68ms
http_reqs......................: 1297244 43221.02723/s
iteration_duration.............: avg=39.08ms min=23.12ms med=38.84ms max=76ms p(90)=43.87ms p(95)=45.71ms
iterations.....................: 7676 255.745723/s
vus............................: 10 min=10 max=10
vus_max........................: 10 min=10 max=10

16 vCPU/16GB RAM/10 VUs - servy

checks.........................: 100.00% ✓ 1407770 ✗ 0
data_received..................: 156 GB 5.2 GB/s
data_sent......................: 172 MB 5.7 MB/s
http_req_blocked...............: avg=2.28µs min=420ns med=1.64µs max=3.96ms p(90)=3µs p(95)=3.94µs
http_req_connecting............: avg=2ns min=0s med=0s max=292.26µs p(90)=0s p(95)=0s
http_req_duration..............: avg=512.52µs min=40.21µs med=177.39µs max=45.65ms p(90)=444.41µs p(95)=623.84µs
{ expected_response:true }...: avg=512.52µs min=40.21µs med=177.39µs max=45.65ms p(90)=444.41µs p(95)=623.84µs
http_req_failed................: 0.00% ✓ 0 ✗ 1407770
http_req_receiving.............: avg=324.43µs min=4.02µs med=26.56µs max=45.31ms p(90)=88.04µs p(95)=147.94µs
http_req_sending...............: avg=4.33µs min=969ns med=3.35µs max=4.11ms p(90)=5.86µs p(95)=7.31µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=183.74µs min=31.45µs med=132.09µs max=8.96ms p(90)=344.35µs p(95)=465.04µs
http_reqs......................: 1407770 46854.747614/s
iteration_duration.............: avg=36.03ms min=5.19ms med=45.31ms max=77.29ms p(90)=48.06ms p(95)=48.95ms
iterations.....................: 8330 277.247027/s
vus............................: 10 min=10 max=10
vus_max........................: 10 min=10 max=10

	`gosherve (old)`	`gosherve`	`servy`
1 vCPU/256MB RAM/1 VUs	86MB/s	587MB/s	547MB/s
2 vCPU/4GB RAM/10 VUs	426MB/s	1.3GB/s	1.7GB/s
16 vCPU/16GB RAM/10 VUs	439MB/s	4.8GB/s	5.2GB/s

Interestingly, it seems that the servy variant actually outperformed gosherve again in these VM-based tests, where it didn’t when run directly on my machine. In any case, the gap is now much smaller between the two, and gosherve still seems to maintain an edge on very small machines.

Note I had to drop the VUs to 1 on the smallest iteration, or I was seeing all three of the variants getting OOM-killed!

I also added a new flake input in the server-bench repo that adds the gosherve based repo again, but pinned to the pre-optimised version.

As such, if you want to try reproducing any of these results and measure old vs. new vs. servy, you just need to do the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Start the devShell for the server-bench project
❯ nix develop github:jnsgruk/server-bench
# Build & run the VM
❯ run-benchvm-vm --daemonize --display none
# Edit the core/memory count as required
❯ vim vm.nix
# Start, and load test a server, choosing your variety
❯ benchvm-test <jnsgruk-go-old|jnsgruk-go|jnsgruk-rust>
# Power down the VM
❯ benchvm-exec poweroff

The results will be gathered in a collection of summary-*.<txt|html|json> files for you to inspect.

Summary #

This was supposed to be a blog about learning Rust, but I’m pleased with where it ended up! This was by no means a particularly in-depth dive into measuring the performance of web servers, focusing almost entirely on the request throughput. There are other factors that could be taken into consideration - such as the resulting memory pressure on the machine, CPU usage, etc. I may come back to this in another post with more comprehensive measurement of both.

Hopefully this post illustrated the power of load testing and profiling tools - even when only used at quite a surface level, and demonstrated how Nix can be used to create robust structures around existing tools to help with such workflows.

I owe a thank you to Frederic from Polar Signals, who gave me a bunch of helpful tips along the way while I was using Parca and profiling gosherve. After my initial improvement in the request handler, Frederic helped me track down and remove the use of fs.ReadFile and further improve the performance of gosherve.

I’m still very early in my Rust development, so if you’ve spotted something horrible in servy, or you’ve got experience with k6 and parca, then reach out and let me know! I’d love to see the creative ways people use these tools.

The performance gap between servy and gosherve is now quite narrow. Close enough, in fact, that they seem to win over one another depending on different conditions in the test environment, and never by a particularly high margin. I haven’t migrated this site over yet, but perhaps I will in the near future. Besides, Fly.io are already kindly limiting the damage that can be done by too many requests for me!

See you next time!

Hot Tub Monitoring with Home Assistant and ESPHome

Tue, 12 Nov 2024 00:00:00 +0000

Introduction #

This year we put a wood-fired hot tub in our back garden. It’s a total indulgence and a real treat. I originally purchased the tub for my wife, but I must admit to having grown quite fond of it myself!

The tub itself is made from Canadian Redwood Cedar. The photo below was taken shortly after it was assembled, so there is still a little bit of leakage which stopped a few days later once the wood had expanded. The 30kW wood burner is the most effective way to heat the tub (which holds around 1700l of water), but there is also a small electric heater in line with the pump & filter which are stashed behind the tub.

There is something wonderfully simple and low-tech about the whole arrangement, but naturally I wanted to get an understanding of the energy usage, and get a reading on the temperature so that I could more accurately set it up for when we wanted to use it. Part of the reasoning for understanding the energy usage was to ensure that the pump and UV filter only fire up when the house is generating enough solar energy to cover it.

A picture paints a thousand words, so you can see what I’m talking about below:

Energy Usage Monitoring #

Starting with the easy part: monitoring the energy usage for each of the components. What you can’t see in the picture above is the following:

An Elecro Vulcan 3kw heater
An evoUV 15w UV clarifier
A Crystal Enterprises CC2513 pump
A Crystal Enterprises CC3030 filter unit

The UV filter, heater and sand filter are all essentially “in line” - when the pump is switched on, water is pulled from the bottom of the tub, through the UV clarifier, sand filter and heater, and then back into the tub (not necessarily in that order!). The heater has a thermostat in it so that it can cut in/out depending on the desired temperature as the water passes through.

In reality, the 3kw heater would take 12-15 hours to heat the tub from scratch, which is why we use the burner that usually takes around 1.5-2 hours. The electric heater is good for maintaining the temperature once the initial heating has been done.

I’ve had quite a lot of success with TP-Link Tapo P110 smart plugs. They’re very cheap, and there is already a competent Home Assistant integration for them. The only downside is that they require a proprietary app for initial setup. After having bought a few of these, I learned about Tasmota, and will probably buy smart plugs compatible with this or similar open source firmwares in the future.

My friends from the Self Hosted Podcast seem to recommend buying pre-flashed smart devices from cloudfree.shop, which I’ll certainly consider next time.

All of the above said - the TP-Link plugs work great for this use-case. I’ve got one for the pump and one for the heater. The pump is configured to run for 2 hours a day during peak daylight hours, and the heater can be toggled as and when I need it. The result, when added to Home Assistant is some neat energy monitoring and the ability to toggle things on and off easily when I need:

In an alternative view that summarises power usage for my home, there is a neat section covering the usage from individual devices (all connected to smart plugs):

As you can see - the heater is incredibly power-hungry, and as a result we hardly use it. In the two hours it ran on the day shown above, it consumed almost twice the energy that my main workstation used all day. We now constrain it’s use to a couple of hours per day in the summer months when solar generation is at its peak - otherwise we just heat the tub with the wood burner as we want.

Temperature Sensor: Initial Solution #

When the tub originally arrived, I looked around for an internet-connected pool thermometer, and was quite surprised at the lack of options. I had assumed this was something relatively common. I ended up ordering an Inkbird IBS-P02R kit.

The kit comprises a small floating thermometer, and a display unit that communicates wirelessly with the thermometer. I picked this particular model because the display unit can be connected to Wi-Fi and then checked on through an app. I had (naively…) assumed that I’d be able to get access through some sort of API, but that turned out not to be the case.

Overall, this solution worked okay. The temperature is reported in 5/10/15 minute intervals according to the configuration, and I had no issues with range/connectivity even though the tub is some distance from my house (and the house is heavily insulated with triple glazing!).

The Inkbird app for iOS leaves a lot to be desired, though. I found the user experience pretty frustrating. The web service was often slow to respond, and it required an annoying number of clicks to get to the information I wanted.

While researching the different ways I could get access to the information, I re-discovered the Inkbird IBS-P01B which is a very similar thermometer to the one I’d bought, but communicates over bluetooth. I’d initially ruled this out since my server is well out of bluetooth range, but I’d also been looking for an excuse to play with an ESP32-based microcontroller…

Temperature Sensor: Homemade Solution #

After a little bit of research I ended up ordering the following:

1x Inkbird IBS-P01B thermometer
1x ESP32-WROOM-32U development board

The former was ordered from AliExpress, and the latter from Amazon. I chose the ESP32-WROOM-32U specifically because it has an external antenna. Given the eventual placement of the ESP32 and the location of the hot tub, I wanted to maximise the chances of establishing a good bluetooth connection.

This was to be my first foray into ESP32/Arduino type development. I’d read (and heard…) lots about ESPHome, and given my goal was to integrate with Home Assistant, this felt like the right route.

Connecting the ESP32 #

The first step was to flash ESPHome onto the device. The development board I bought comes with a CP2012 USB to UART bridge on the board, so connecting the device to my machine was as simple as plugging it in with a MicroUSB cable.

Before the device can be flashed, it needs to be put into programming mode. In my case, the board has a handy BOOT button. The process for entering programming mode was therefore (starting with the device unplugged and powered down):

Press and hold the BOOT button
Connect the ESP32 over USB
Wait a few seconds
Release the BOOT button

Other development boards may not have a button, but all it really does is bridge GPIO0 and GND on the board which can be done with a wire, too. There are good docs on the connection process on the ESPHome website.

ESPHome Support #

ESPHome doesn’t natively support the IBS-P01B, but after some reading it seemed that the underlying messaging format/protocol is similar enough to the IBS-TH1 and IBS-TH2 that the same configuration can be used.

I also came across this blog post where the author had manually created an ESPHome configuration that appeared to work, and with the exact same hardware.

Nonetheless - to my mind, simpler is better most of the time, so I set about creating a simple configuration using the supported ESPHome platform.

ESPHome Dashboard #

There are a few different ways to get started with ESPHome. You can “install” using Docker, which seems to be the preferred method (passing through /dev/ttyUSBx to the container for USB connectivity), or manually using Python packages.

I tried to follow the instructions to get started with Home Assistant, but it seems this is not compatible with “unsupervised” Home Assistant servers like mine.

The esphome package is readily available in nixpkgs, so I was able to get started and fire up the dashboard like so:

❯ nix run unstable#esphome -- dashboard .
2024-11-12 14:47:19,072 INFO Starting dashboard web server on http://0.0.0.0:6052 and configuration dir ....

Browsing to http://localhost:6052 yielded me the following page:

Creating a Firmware #

I proceeded to create a new device, and entered some basic information about my Wifi network:

This is where things got interesting (for me, at least!). It turns out that the ESPHome dashboard makes use of the Web Serial API to program the ESP32 chip over USB from the browser. Given that my board was already connected and in programming mode, this was pretty simple!

The first time I tried this, I got a permission error. After looking through the messages in my kernel’s ring buffer with journalctl -k, and looking at the permissions on the /dev/ttyUSB0 device, it seemed likely that the issue was the root:root ownership with limited permissions. I solved this by running the following:

1
2


sudo chown root:users /dev/ttyUSB0
sudo chmod g+rwx /dev/ttyUSB0

This changed the group of the serial device to the same group as my user, then gave the group read, write and execute permissions. I retried in the browser and the device was flashed with the new firmware, connecting it to my network! Once the device was online, it appeared as such in the ESPHome dashboard, and I cooked up the following configuration by studing the docs on the inkbird_ibsth1_mini sensor component:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44


substitutions:
 name: esphome-web-caf3b8
 friendly_name: Tub Monitor

esphome:
 name: ${name}
 friendly_name: ${friendly_name}
 min_version: 2024.6.0
 name_add_mac_suffix: false
 project:
 name: esphome.web
 version: dev

esp32:
 board: esp32dev
 framework:
 type: arduino

web_server:
improv_serial:
logger:
api:
ota:
 - platform: esphome

wifi:
 # Pulled in from a separate secrets.yaml
 ssid: !secret wifi_ssid
 password: !secret wifi_password

esp32_ble_tracker:
 scan_parameters:
 active: true
 continuous: true
 duration: 1min

sensor:
 - platform: inkbird_ibsth1_mini
 # Bluetooth MAC address of my IBS-P01B
 mac_address: <redacted>
 temperature:
 name: "Tub Temperature"
 battery_level:
 name: "Tub Monitor Battery"

A few points on the above:

The device setup wizard created a secrets.yaml which can hold secrets to be referenced with the !secret <secret name> syntax. This is the same format as used by Home Assistant.
Specifying web_server: enables a small embedded web server that can be used to check status (see below!).
I had to specify the MAC address of the pool thermometer, which I collected using the Bluetooth Inspector app for iOS.

I clicked Install in the top right of the window, and the firmware was compiled and flashed to the device! This time I was able to flash over Wifi since the device was now connected to my network as a result of the onboarding process:

After selecting “Wirelessly”, and waiting for the firmware to be flashed, the logs shortly started flowing in:

To double check things were working, I then browsed to the IP address of the ESPHome device, and was greeted with a simple page that shows the current state of the device:

Adding to Home Assistant was as simple as adding a new integration of type ESPHome and specifying the IP address of the device!

3D Printing an Enclosure #

The final part of my build was to create a small enclosure for the device that would house the antenna correctly. I found a nice model on Printables, which printed pretty fast on my BambuLab X1 Carbon printer.

The photo below shows the device in place on the window sill, with the hot tub just visible in the background:

Home Assistant Dashboard #

With all this in place, all that remained was tying the information into my Home Assistant dashboard:

The above shows the hot tub temperature, as well as the battery level of the IBS-P01 sensor. Beneath, I’ve added basic controls and measurements available through the Tapo Home Assistant integration, which allows me to quickly toggle the pump and heater, and see their daily energy usage at a glance.

If you look closely, you’ll see the room temperatures resulting from the Roth underfloor heating integration I wrote about previously.

Summary #

We’re super pleased with our tub - it was built by Andy from Rustic Tubs. If you’re considering one and you’re in the UK, you could do a lot worse. It’s beautifully crafted, and Andy was really helpful and communicative throughout the process.

This was my first foray into ESP32-based projects, and I was pleasantly surprised with how seamless the process was. The ESPHome docs were clear, and the no-hassle flashing through the browser was a nice way to get started.

I hope this post is useful to people who are just getting started, and feel free to reach out if I’ve missed something!

Writing a Home Assistant Core Integration: Part 2

Wed, 16 Oct 2024 00:00:00 +0000

Introduction #

In my last post I described the first steps toward creating a new Home Assistant integration for the underfloor heating system in my house. In that post I outlined in detail how I set about creating a Python client for the API provided by the underfloor heating controller vendor.

In this post, I’ll describe the development setup, project structure and contribution process for building and landing a Home Assistant Core integration. I don’t consider myself an expert here, but I’ve documented my journey here in the hope that my experience might be useful to potential future contributors.

The finished integration can be seen in the Home Assistant docs under the name touchline_sl, and the code can be found on Github.

Development Setup #

The Home Assistant documentation recommends the use of Visual Studio Code with a devcontainer. This was a very quick way to get a working environment up and running, especially given that I already use Visual Studio Code for most of my programming, so I was immediately familiar.

The repository provides some tasks to help get started, including a task named Run Home Assistant Core, which takes care of setting up the runtime environment, installing dependencies and starting the server. Neat!

There are also a set of pre-commit hooks set up to ensure you don’t make any common mistakes, accidentally violate the formatting/static-typing rules for the project, forget to update requirements.txt files, etc.

This turned out to be a really nice way to get started, and if you’re new to Home Assistant Core development, I’d recommend giving this “batteries-included” approach a go. If it’s not for you, the project provides manual setup instructions too.

Integration Basics #

According to the documentation:

[An] integration is responsible for a specific domain within Home Assistant. Integrations can listen for or trigger events, offer actions, and maintain states.

Where a domain is…

a short name consisting of characters and underscores. This domain has to be unique and cannot be changed.

Err, right… while this is an accurate statement, it’s perhaps not the most enlightening for the budding new integration developer! At their core, integrations are Python modules that take information about a given system (like an underfloor heating system, or a smart plug, or a light bulb) and represent information about them in a format compatible with one of Home Assistant’s archetypes for devices/sensors/platforms/entities.

In this case, we’ll be representing Climate Entities, which have the sort of properties you might expect - target_temperature, current_humidity, current_temperature, etc.

To use Home Assistant’s terms, in my setup:

The platform is the underfloor heating system
The platform consists of some devices, in this case the physical thermostats in each room of my house
The devices each represent one or more climate entities (humidity, temperature, etc.)

File Structure #

The basic file structure can be laid down with some scaffold tooling, but even in its finished state, my integration doesn’t have many files:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


.
├── __init__.py # the component file
├── climate.py # ties info from the api into home assistant terms
├── config_flow.py # defines the fields/flow for integration config
├── const.py # constants used across the integration
├── coordinator.py # data update coordinator
├── manifest.json # defines project dependencies and metadata
├── strings.json # defines strings displayed in various ui elements
└── translations # a directory containing one file per language
 └── en.json # english translation of strings.json

And to give an idea of the scale of the project in its completed form:

----------------------------------------------------------------------
Language files blank comment code
----------------------------------------------------------------------
Python 5 71 27 215
JSON 3 0 0 82
----------------------------------------------------------------------
SUM: 8 71 27 297
----------------------------------------------------------------------

`manifest.json` #

Starting with the most simple first! The manifest.json describes the integration: what its name is, where its documentation is found, who owns the code and the libraries it depends on:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


{
 "domain": "touchline_sl",
 "name": "Roth Touchline SL",
 "codeowners": ["@jnsgruk"],
 "config_flow": true,
 "documentation": "https://www.home-assistant.io/integrations/touchline_sl",
 "integration_type": "hub",
 "iot_class": "cloud_polling",
 "requirements": ["pytouchlinesl==0.1.8"]
}

Here I selected cloud_polling as the iot_class, because my integration reaches out periodically to the API, polling for new information. You’ll note also that the integration requires my pytouchlinesl library in order to run.

`init.py` #

Typically for integrations, this file defines how to setup the integration, and how to unload it. You can see the full source of my implementation on Github.

I define an async_setup_entry() method which takes care of:

Establishing a coordinator per TouchlineSL module
Performing the initial hydration of data
Registering each TouchlineSL module as a device in Home Assistant’s Device Registry, which is where Home Assistant “keeps track of devices”

There is a shortened, annotated version of the method below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


async def async_setup_entry(hass: HomeAssistant, entry: TouchlineSLConfigEntry) -> bool:
 """Set up Roth Touchline SL from a config entry."""
 account = TouchlineSL(...)

 coordinators: list[TouchlineSLModuleCoordinator] = [
 TouchlineSLModuleCoordinator(hass, module)
 for module in await account.modules()
 ]
 # ...
 device_registry = dr.async_get(hass)
 # Create a new Device for each coorodinator to represent each module
 for c in coordinators:
 module = c.data.module
 device_registry.async_get_or_create(
 config_entry_id=entry.entry_id,
 identifiers={(DOMAIN, module.id)},
 name=module.name,
 # ...
 )

 entry.runtime_data = coordinators
 await hass.config_entries.async_forward_entry_setups(entry, PLATFORMS)
 return True

Before returning async_setup_entry() invokes async_forward_entry_setups(). This ensures that async_setup_entry() in climate.py is called to ensure each of the climate entities is registered.

The PLATFORMS variable is a list of platform types that the integration supports, in this case a single-item list containing just Platform.CLIMATE, which is how async_forward_entry_setups knows to invoke the async_setup_entry() method in climate.py:

1
2
3
4


from homeassistant.const import Platform
# ...
PLATFORMS: list[Platform] = [Platform.CLIMATE]
# ...

To initialise the integration, the user must authenticate with the Roth API so that module details are fetched before constructing the coordinator. Thus, before this code is executed, the user must go through the config flow…

`config_flow.py` #

Despite the final implementation being quite simple, this is probably one of the areas I found most challenging to get right. There are some docs but they only scratch the surface, and implementations in other integrations seem to vary quite dramatically (mostly depending on when they were written).

I went through a few iterations of this config flow, mostly because I had originally implemented the ability to select a specific module from the user’s account. The review process guided me toward simply logging into the account, then enrolling each of the modules associated with the account - since users can always disable entities they don’t wish to manage in Home Assistant.

The config_flow.py defines which input fields need to be presented to the user, and then passes on the relevant information needed to set up the integration. In my implementation, the code:

Prompts the user for their username and password
Authenticates with the service and fetches the user’s unique ID
Registers that unique ID, aborting if the specified account has already been used
Creates a config entry in Home Assistant that stores the user’s credentials

The config flow also has some basic error handling that can distinguish the difference between poor credentials, networking issues, etc. The full implementation can be seen on Github, but the important parts are highlighted in the following snippet:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


# ...
class TouchlineSLConfigFlow(ConfigFlow, domain=DOMAIN):
 # ...
 async def async_step_user(
 self, user_input: dict[str, Any] | None = None
 ) -> ConfigFlowResult:
 # ...
 if user_input is not None:
 try:
 account = TouchlineSL(
 username=user_input[CONF_USERNAME],
 password=user_input[CONF_PASSWORD],
 )
 # Use the credentials to fetch unique user id
 await account.user_id()
 except RothAPIError as e:
 # ...
 else:
 # Set unique ID, abort setup if already used
 unique_account_id = await account.user_id()
 await self.async_set_unique_id(str(unique_account_id))
 self._abort_if_unique_id_configured()

 # Create a config entry containing the user's credentials
 return self.async_create_entry(
 title=user_input[CONF_USERNAME], data=user_input
 )

 return self.async_show_form(
 step_id="user", data_schema=STEP_USER_DATA_SCHEMA, errors=errors
 )

What caught me out was how the fields are given titles/descriptions. These attributes are all configured in the strings.json, where the config map contains keys for each of the config “steps”.

The code above defines a step named user, since the method name is async_step_user. The step’s name, description and input fields are defined in the strings.json:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


{
 "config": {
 "flow_title": "Touchline SL Setup Flow",
 "error": {
 "cannot_connect": "[%key:common::config_flow::error::cannot_connect%]"
 // ...
 },
 "step": {
 "user": {
 "title": "Login to Touchline SL",
 "description": "Your credentials for the Roth Touchline SL mobile app/web service",
 "data": {
 "username": "[%key:common::config_flow::data::username%]",
 "password": "[%key:common::config_flow::data::password%]"
 }
 }
 },
 "abort": {
 "already_configured": "[%key:common::config_flow::abort::already_configured_device%]"
 }
 }
 // ...
}

The values displayed to the user are pulled from the translation files at runtime depending on their language configuration. In my integration, the corresponding translations/en.json contains the following fields that map to those defined above:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


{
 "config": {
 "flow_title": "Touchline SL Setup Flow",
 "error": {
 "cannot_connect": "Failed to connect"
 // ...
 },
 "step": {
 "user": {
 "data": {
 "password": "Password",
 "username": "Username"
 },
 "description": "Your credentials for the Roth Touchline SL mobile app/web service",
 "title": "Login to Touchline SL"
 }
 },
 "abort": {
 "already_configured": "Device is already configured"
 }
 }
 // ...
}

In hindsight, this is covered in the docs, but it definitely didn’t click with me when I was going through it, so it seemed worth calling out! The net result of this setup is a configuration dialog that looks like so:

`coordinator.py` #

This was an addition I made during the review process (more on that later), and appears to be the preferred way to implement the fetching of data from upstream APIs. By implementing a DataUpdateCoordinator, Home Assistant can ensure that a single coordinated poll happens across all entities managed by an integration. If an integration manages many entities for which it needs to fetch/update details, the coordinator helps ensure that the API is called only as often as is needed.

The coordinator class is very simple: it defines a single method _async_update_data which returns the data for the device its coordinating. As the developer, you can specify the type of the data returned by the coordinator. I chose to represent this as a Python dataclass:

1
2
3
4
5


@dataclass
class TouchlineSLModuleData:
 module: Module
 zones: dict[int, Zone]
 schedules: dict[str, GlobalScheduleModel]

The coordinator is initialised with some basic information such as a name and an update interval:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# ...
class TouchlineSLModuleCoordinator(DataUpdateCoordinator[TouchlineSLModuleData]):
 def __init__(self, hass: HomeAssistant, module: Module) -> None:
 super().__init__(
 hass,
 logger=_LOGGER,
 name=f"Touchline SL ({module.name})",
 update_interval=timedelta(seconds=30),
 )

 self.module = module

The _async_update_data method then queries the API, and returns data in the newly defined format:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


async def _async_update_data(self) -> TouchlineSLModuleData:
 try:
 zones = await self.module.zones()
 schedules = await self.module.schedules()
 except RothAPIError as error:
 # ...
 # Return the data using our dataclass from above
 return TouchlineSLModuleData(
 module=self.module,
 zones={z.id: z for z in zones},
 schedules={s.name: s for s in schedules},
 )

You can see the full implementation on Github.

`climate.py` #

And finally, on to the business logic of tying fields from the upstream API into the relevant attributes in Home Assistant!

The first task handled by this file is registering each of the climate entities by iterating over each zone, in each coordinator’s module:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# ...
async def async_setup_entry(
 hass: HomeAssistant,
 entry: TouchlineSLConfigEntry,
 async_add_entities: AddEntitiesCallback,
) -> None:
 """Set up the Touchline devices."""
 coordinators = entry.runtime_data
 async_add_entities(
 TouchlineSLZone(coordinator=coordinator, zone_id=zone_id)
 for coordinator in coordinators
 for zone_id in coordinator.data.zones
 )
# ...

Home Assistant entities have well-defined APIs - the docs for climate entities show the supported attributes and their data types. I used a combination of the docs, and the source code to establish how to implement my ClimateEntity, which boiled down to the following interface:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


class TouchlineSLZone(CoordinatorEntity[TouchlineSLModuleCoordinator], ClimateEntity):
 # Construct a Touchline SL climate zone.
 def __init__(self, coordinator: TouchlineSLModuleCoordinator, zone_id: int) -> None:
 # Handle updated data from the coordinator.
 @callback
 def _handle_coordinator_update(self) -> None:
 # Return the device object from the coordinator data.
 @property
 def zone(self) -> Zone
 # Report if the device is available.
 @property
 def available(self) -> bool
 #Set new target temperature.
 async def async_set_temperature(self, **kwargs: Any) -> None
 # Assign the zone to a particular global schedule.
 async def async_set_preset_mode(self, preset_mode: str) -> None
 # Populate attributes with data from the coordinator.
 def set_attr(self) -> None:

Arguably the most important part here is set_attr(). which takes care of mapping fields from the objects provided by my pytouchlinesl library to attributes on the climate entity:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


def set_attr(self) -> None:
 """Populate attributes with data from the coordinator."""
 schedule_names = self.coordinator.data.schedules.keys()

 self._attr_current_temperature = self.zone.temperature
 self._attr_target_temperature = self.zone.target_temperature
 self._attr_current_humidity = int(self.zone.humidity)
 self._attr_preset_modes = [*schedule_names, CONSTANT_TEMPERATURE]

 if self.zone.mode == "constantTemp":
 self._attr_preset_mode = CONSTANT_TEMPERATURE
 elif self.zone.mode == "globalSchedule":
 schedule = self.zone.schedule
 self._attr_preset_mode = schedule.name

The full implementation is on Github.

Testing #

Testing is (understandably) a little complicated. The requirements for landing code for an integration stipulate that you must include unit tests for any config/options flows. In my case, this meant the following:

Implementing a mock TouchlineSL client fixture to ensure that the unit tests don’t try to reach out to the real Roth API
Implementing a mock config entry fixture
A unit test for a successful config flow execution
A parametrised unit test for unsuccessful config flow due to possible exceptions when hitting the API
A unit test to ensure that multiple config flows resulting in the same user ID fail

To me this feels like the bare minimum, and unfortunately doesn’t really provide any confidence that the integration actually functions correctly. I’m hoping to improve this in the future, but for now further testing has been manual.

Docs & Brand #

As I was creating the Pull Request to contribute my integration, I was prompted by the template to link to further PRs that added the documentation and brand assets for my integration. I used the existing touchline docs as a template and modified them for my integration. The docs pull request added a single file touchline_sl.markdown containing 29 lines, which results in some nicely rendered docs on the Home Assistant website:

There were already branding assets for Roth as part of the touchlinesl integration that was previously merged. Based on some review feedback, I created a new Integration Brand named roth, with which both the touchline and touchline_sl integrations are associated. This has the nice effect of grouping them when setting up a new integration:

Contribution Process #

I submitted my first efforts for review in home-assistant/core#124557. The checklist guided me nicely through what needed to be done, and overall the process went pretty smoothly, and pretty quickly (despite the 117 comments!). The process took a little under two days in total. I was also fortunate with my timing, since my code landed the day before the next beta release was cut, so it shipped relatively quickly.

I’d like to offer my sincere thanks to @joostlek who not only reviewed my code extremely quickly, but took the time to explain things to me both in the Github PR, but also by proactively reaching out to me on the Home Assistant Discord, which I really appreciated.

I got a lot of feedback, which I expected. This was my first attempt at writing code for Home Assistant, and it’s a pretty large and well established project. I do think the project could do with some better developer documentation, which would dramatically reduce the burden of effort on reviewers and give contributors a better chance of “getting it right”.

One might argue I could have done more reading and more research - though I spent a good amount of time reading both the documentation and the source code of other integrations. I’m not sure I’d ever have reached the conclusions that @joostlek kindly nudged me toward.

Overall, my implementation was more brief, more simple and more efficient as a result of the review process, and based on my experience I’d advocate for having a go if you’ve been on the fence! Often submitting code to such a project can be daunting, but as with my experience when contributing to nixpkgs, if you go in with an open mind you’re sure to learn something from the process.

Results #

I’m really pleased with the results. However obtuse the developer experience felt at times, there is no denying that what I was able to get from my 297 lines of implementation code is quite staggering. I was super impressed that just by following the conventions I was able to get such intuitive controls, a full graphable history of temperatures and such a wide variety of ways to display the information in my various dashboards.

Once set up, you’re able to get a view of all the different zones imported by the module (the names of each zone are pulled from the upstream API if the zones are named, and each can be associated with a given area in Home Assistant):

The default dashboard displays thermostat controls for each zone. These allow you to see the current and target temprature, as well as adjust the target temperature if you need to:

Expanding the thermostat cards gives you a more detailed view, showing the mode and the “Preset”. My implementation maps “Presets” to “Global Schedules” configured in the Roth module:

Summary #

And that’s a wrap! I learned a bunch writing this integration, and the resulting user experience is quite a lot better than I get with the default Roth application.

My next mission is to use the information from these climate entities to automate opening the Velux windows in the roof when things get warm in the summer, but I’ve got a lot to learn about Home Assistant in the mean time!

Writing a Home Assistant Core Integration: Part 1

Wed, 11 Sep 2024 00:00:00 +0000

Introduction #

Back in March, my family and I moved into a new home. It’s a modern construction which came with solar panels (and associated inverter/battery storage), and uses an air source heat pump to heat the house with underfloor heating. Being a new renovation, nearly all of the appliances and components in the house have a form of internet connectivity (some more useful than others!).

Since day 1, I’ve been hoping to consolidate all of the various applications, data feeds and functions into one single place. I’ve been a long-time listener to the Self Hosted podcast, which often extols the virtues of Home Assistant. I’ve got no prior experience with Home Assistant, but for the last three months I’ve been running it on my home server, with a collection of custom integrations and hacks that enable me to control the underfloor heating and solar inverter.

The underfloor heating controller is a Roth Touchline SL system. In my set up, there is a single “module” which represents my house, and a number of “zones” which represent different rooms.

There was unfortunately no integration for this system in Home Assistant - there is one for the previous generation “Roth Touchline”, but this appears to function over the LAN, whereas the Touchline SL system is controlled over the internet using their API.

After reading the source code of a few other climate integrations in Home Assistant, it became clear to me that the first step was to create a Python client for the API which could be used in the integration.

This post will cover the design, implementation and limitations of the library I wrote: pytouchlinesl. If you came here to read about writing code for Home Assistant, you’ll have to wait for the next post! 😉

Designing the library #

Upstream API #

Usually, one would interact with a Roth Touchline SL system through their mobile apps, or through their online portal. The mobile app seems to be quite a lightweight wrapper around the web application, and I’ve not been able to detect any difference in functionality.

A bit of searching uncovered that Roth also provide an API for the Touchline SL system, and an OpenAPI spec. This made the process significantly easier, though there are some discrepancies in what is documented compared with how the API actually behaves. It feels to me like the API may have evolved, and the documentation has remained static - or perhaps it was always inaccurate? Either way, I spent quite a bit of time manually fuzzing the API to work out the correct set of parameters for some endpoints.

I also studied the web application using the Chrome Dev Tools. Of all the endpoints documented, it seemed like I’d only need the following:

POST /authentication: authenticates with the API, taking a username and password, and returning a token
GET /users/{user_id}/modules: returns a list of modules associated with the user
GET /users/{user_id}/modules/{module_udid}: returns details of a specific module (zones, schedules, etc.)

There is a slight awkwardness here, in that the last endpoint returns all of the data - for all zones, all schedules, etc. This feels inefficient, but I couldn’t find a way of getting information about a specific zone, or a specific schedule. The web application seems to rely upon polling the (undocumented) endpoint GET /users/{user_id}/modules/{module_id}/update/data/parents/[]/alarm_ids/[]/last_update/{timestamp}, which delivers deltas in the data since a given timestamp. This is useful in the context of the app because it can request the full dataset once, then request only changes from that point onwards, keeping the app state up to date without requesting the whole dataset.

Making changes to the configuration of zones and their temperatures is also fragmented. In essence, one can:

Set a zone to a constant temperature: POST /users/{user_id}/modules/{module_udid}/zones
Place a zone on a global schedule: POST /users/{user_id}/modules/{module_udid}/zones/{zone_id}/global_schedule
Place a zone on a local schedule: POST /users/{user_id}/modules/{module_udid}/zones/{zone_id}/local_schedule

The first is self-explanatory, enabling a zone to be set to a constant temperature (19.0C, for example). Touchline SL modules also support “schedules” which contain time periods for the specified zones to reach certain temperatures. In the case of a “Global Schedule”, multiple zones can be assigned, while a “Local Schedule” is specific to a single zone. The awkwardness in the API here is that to “add” a zone to a global schedule, you must re-specify the entire schedule, and specify all of the zones that should be on the schedule…

Basic Requirements #

In order to fulfil the basic functionality of my (future) Home Assistant integration, I limited the requirements of the first version to:

Authenticate with the API using a username and password
List modules associated with the account
Get a specific module
Get a specific zone
Get a list of global schedules
Get a specific global schedule
Assign a constant temperature to a zone
Assign a zone to a specific global schedule

I don’t use local schedules in my system, so I’ve omitted them for now, though updating the library to support them would be trivial.

Outline design/experience #

With those requirements in mind, I came up with a rough sketch of how I’d like the library to behave:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


tsl = TouchlineSL(username="foo", password="bar")
module = await tsl.module(id="deadbeef")

lounge = module.zone_by_name("Lounge")
kitchen = module.zone(1234)

# Properties should be available such as:
# lounge.current_temperature
# kitchen.humidity

await kitchen.set_temperature(temperature=20.0)

living_spaces = await tsl.schedule_by_name(schedule_name="Living Spaces")
await lounge.set_schedule(living_spaces.id)

And from that came a reasonable outline of the API for the library:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


class TouchlineSL:
 # Construct a class that represents a Touchline SL account
 def __init__( self, *, username: str | None = None, password: str | None = None)
 # Get a list of modules associated with the account
 async def modules(self) -> list[Module]
 # Get a specific module, by ID
 async def module(self, *, module_id: str) -> Module | None

class Module:
 # Get a list of zones from the module, optionally including disabled zones
 async def zones(self, *, include_off: bool = False) -> list[Zone]
 # Get a specific zone, by ID
 async def zone(self, zone_id: int) -> Zone | None
 # Get a specific zone, by name
 async def zone_by_name(self, zone_name: str) -> Zone | None
 # Get a list of global schedules
 async def schedules(self) -> list[Schedule]:
 # Get a specific schedule, by ID
 async def schedule(self, schedule_id: int) -> Schedule | None
 # Get a specific schedule, by name
 async def schedule_by_name(self, schedule_name: int) -> Schedule | None

class Zone:
 # Get the schedule the zone is assigned to
 def schedule(self) -> Schedule | None
 # Set the zone to a constant temperature
 async def set_temperature(self, temperature: float)
 # Assign the zone to a specific schedule
 async def set_schedule(self, schedule_id: int)

Note that after reading the code from other climate integrations in Home Assistant, it became clear to me that they favour the use of async libraries, and thus my library was designed to use asyncio from the start.

Python tools/libraries #

There are a couple of things I’ve found tiresome about Python over the years, but things do seem to be looking up. I’ve always found the package management and distribution to be awkward, and I can’t be the only one if the number of projects looking to target that problem is anything to go by (e.g. poetry, rye, uv, pdm, etc.).

Part of this seems to come from fractures in the community itself - there (still!) appears to be disagreements surrounding PEPs such as PEP-621 which introduced pyproject.toml as a way of managing project metadata and dependencies, with the maintainers of some high-profile and widely adopted libraries refusing to adopt it.

That said, there are a couple of things I’ve been meaning to try in anger, and this project was a good opportunity to do so:

`uv` #

Developed by Astral, uv is the “new shiny” at the time of writing, and I can understand why. Pitched as “Cargo, but for Python”, it aims to solve a myriad of problems in the Python ecosystem. uv can handle the download/install of multiple Python versions, the creation of virtual environments, running Python tools in a one-off fashion (like pipx), locking dependencies deeply in a project (by hash) and still maintains a pip compatible command-line experience with uv pip. To add to all of that, it’s ridiculously fast; on a couple of occasions I’ve actually found myself wondering if it did anything when installing dependencies for large projects, because it’s so much faster than I’m used to.

`pydantic` #

Pydantic is a data validation library for Python. It’s entirely driven by Python’s type-hints which means that you get nice integration with language servers. Pydantic allows you define data models in native Python, but emit standard JSON Schema docs for models. It’s integrated quite widely across the Python ecosystem, and to me feels like it bridges the gap between what I hoped type annotations would do for Python, and what they actually do in reality!

`ruff` #

I’ve been using this one for at least the last year for all my Python linting and formatting needs, but I still feel it deserves a call out. There have been a couple of small changes to command line API and such along the way, but overall I’ve found it to be a dramatic improvement over my last setup - which comprised of black, isort, flake8 and a pile of plugins. I had no particular beef with black, but I find flake8’s lack of pyproject.toml support irritating, and grew tired of plugins failing as flake8 released new versions.

In my experience, ruff is stupid fast, and because it ships flake8-compatible rules for all of the plugins I was using as one bundle, they never break. It’s also nice to just have one tool to use everywhere. If you’re interested, you can see how I configure ruff in the pyproject.toml.

Implementation #

With a basic design in mind, and tooling ready to go, I set about building the library itself. It was now time to reconcile my intended design with the realities of the provisions made by the upstream API.

I mentioned previously that the only useful endpoint for getting information about zones/schedules would in fact return all of the data for a given module. Too many calls to this endpoint would likely result in poor performance, so I wanted to introduce some basic caching along the way.

Client implementation #

For the underlying API client implementation, I opted for the following:

BaseClient: a class which inherits from Python’s abc.ABC. This enables the creation of multiple client implementations by defining of the set of methods/properties that any client interacting with the Roth API should define. This decision is primarily to support testing through dependency injection, rather than mocking with patches (more details on that later).
RothAPI: a concrete implementation of the BaseClient abstract class. It is here that I built the actual implementation of the client which handles authentication, GETing and POSTing data, caching, and marshalling API responses into the correct types (defined with Pydantic).

Also included in the client package is the models package. The models package contains (mostly) auto-generated Pydantic models based on real-life responses I got from the API. This is really a function of laziness, but was a convenient way to get type annotated models for the responses I was receiving from the API. Each time I hit a new endpoint, I took the JSON result and did a quick conversion with https://jsontopydantic.com/, before manually adjusting names and updating some fields with Literals.

Caching #

I mentioned earlier that I wanted to implement some basic caching. While I am aware of various plugins for aiohttp (and other request libraries) that could handle this for me, my requirements were quite simple, so I chose to just build it into the library. In this case, caching is implemented on the Module class. This is because the large blob of data that is requested to populate details about a module and its zones/schedules is requested per module.

The caching works like this:

The Module class has “private” attributes named _raw_data and _last_fetched:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


class Module:
 def __init__(
 self,
 *,
 client: BaseClient,
 module_data: AccountModuleModel,
 cache_validity: int = 30,
 ):
 # ...

 # Raw data about the zones, schedules, tiles in the module
 self._raw_data: ModuleModel
 # Unix timestamp representing the last time the _raw_data was fetched
 self._last_fetched = 0
 self._cache_validity = cache_validity

 # ...

There is only one method on this class that calls the underlying client, and that’s another “private” method named _data. This method takes an optional refresh keyword argument, which forces the _raw_data attribute to be updated, but by default will only fetch data if the cached data has expired (after the number of seconds specified in self._cache_validity). If refresh is false, and the cache hasn’t expired, it simply returns the stored raw data:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


async def _data(self, *, refresh: bool = False) -> ModuleModel:
 """Get the raw representation of the module from the upstream API.

 If the data has never been fetched from upstream, or the data is older
 than the cache validity period, then the data is refreshed using the
 upstream API.

 Args:
 refresh: (Optional): Force the data to be refreshed using the API.
 """
 if refresh or (round(time.time() * 1000) - self._last_fetched) > self._cache_validity:
 self._raw_data = await self._client.module(self.id)
 self._last_fetched = round(time.time() * 1000)

 return self._raw_data

Each of the public methods (zones(), zone(), schedule(), etc.) access the raw data through the _data() method, and pass through the refresh flag which is exposed. This means that any developer consuming this library can chose to live with the caching, or override it and force a refresh like so:

1
2
3
4
5
6
7


tsl = TouchlineSL(username="foo", password="bar")
module = await tsl.module(id="deadbeef")

# Request a zone, accepting cached data (default)
zone = await module.zone_by_name("kitchen")
# Or force the data to be refreshed using the upstream API
zone = await module.zone_by_name("kitchen", refresh=True)

Testing, CI & Publishing #

Testing #

In the previous section, I described how I’d set up an abstract base class for the API client, then created an implementation of that for the actual Roth API. One of the main reasons I like this pattern for API clients is that it simplifies testing and reduces the need for monkey patching or traditional mocking (yes, I know the fake API client is sort of a mock…).

Because TouchlineSL can optionally be constructed with any client that implements BaseClient’s abstract base class, it’s trivial to implement a fake API backend that returns fixture data to be used in testing. In this case, the fixtures are stored as JSON files in the repository, and contain real life responses received from the API.

The FakeRothAPI returns the sample data for each of the methods defined in the abstract class. The following is a partial extract from the fake client code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


# ...
from pytouchlinesl.client import BaseClient
from pytouchlinesl.client.models import AccountModuleModel, ModuleModel

data_dir = Path(os.path.realpath(__file__)).parent / "sample-data"


class FakeRothAPI(BaseClient):
 def __init__(self):
 self._user_id = 123456789
 self._token = "deadbeef"

 async def user_id(self) -> int:
 return self._user_id

 async def authenticated(self) -> bool:
 return True if self._token else False

 async def modules(self) -> list[AccountModuleModel]:
 with open(data_dir / "modules.json", "r") as f:
 data = json.load(f)

 return [AccountModuleModel(**m) for m in data]

#...

From there, I defined a number of test fixtures using pytest’s @pytest.fixture decorator, which provide an initialised TouchlineSL instance, backed by the fake client, a Module instance, and a Zone instance.

From there, the tests are fairly simple. Beyond injecting the fake client, there is no mocking required, which in my opinion keeps the test code much easier to read and understand. It also means I could focus more of my energy on validating the logic I was testing, rather than worrying about how patching might interact with the rest of the code. I’ve felt bad about mocking for years, and I think the clearest articulation I’ve seen is “Don’t Mock What You Don’t Own” in 5 Minutes.

One aspect of the test suite I don’t love is the use of time.sleep in certain tests. This is because my caching implementation relies on reading a timestamp to decide on whether it should refresh data. In general I steer away from sleeps in tests, as they’re often used to mask an underlying non-determinism, but in this case it felt a reasonable trade-off, given that I’m testing a time-based functionality.

CI #

I wanted to ensure that any pull requests were tested, and that they conform to the project’s formatting/linting rules. Since the project is hosted on Github, I used Github Actions for this. The pipeline for this project is pretty simple, it lints and formats the code with ruff failing if any files were changed by the formatter or any of the linting rules were violated.

Finally, uv is used to run pytest across a matrix of supported Python versions. I could have used uv to handle the download and install of different Python versions too, but the setup-python actions has served me perfectly well in the past.

Publishing #

Publishing is taken care of in CI. I don’t like having to remember the magic incantation for building, authenticating and publishing locally. I want the process to be as consistent and as transparent as possible for the people consuming the project.

What was new to me this time was publishing to PyPI with a “Trusted Publisher” setup. To quote their docs:

“Trusted publishing” is our term for using the OpenID Connect (OIDC) standard to exchange short-lived identity tokens between a trusted third-party service and PyPI. This method can be used in automated environments and eliminates the need to use manually generated API tokens to authenticate with PyPI when publishing.

Rather than setting up Github Actions to hold an API token in a Secret, there is some automation that links a Github project to a PyPI project. The project doesn’t even have to be previously published on PyPI to get started, and new projects can be configured as “Pending”, then published to for the first time from your CI system of choice 🚀.

I configured Github Actions to trigger the release of pytouchlinesl on new tags being pushed:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


# ...
publish:
 name: Build and Publish to PyPI
 runs-on: ubuntu-latest
 needs:
 - tests

 # Trusted publisher setup for PyPI
 environment:
 name: pypi
 url: https://pypi.org/p/pytouchlinesl
 permissions:
 id-token: write

 steps:
 - name: Checkout the code
 uses: actions/checkout@v4

 - name: Install `uv`
 run: |
 curl -LsSf https://astral.sh/uv/install.sh | sh

 - name: Build the package
 run: |
 uvx --from build pyproject-build --installer uv

 - name: Publish to PyPi
 uses: pypa/gh-action-pypi-publish@v1.10.0

Contributing to `nixpkgs` #

Finally, my Home Assistant server runs NixOS, so in order for the integration to be packaged easily in the future, I created a small PR to include pytouchlinesl in nixpkgs. The original PR went through pretty quickly - thanks to @drupol for the fast review!

Since then I’ve made a couple of minor version bumps to the library as I discovered small issues when building the integration, but the final derivation is quite compact (see below). The Python build tooling in Nix is quite mature at this point, and I gained a fair bit of experience using it when packaging snapcraft and charmcraft.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32


#...

buildPythonPackage rec {
 pname = "pytouchlinesl";
 version = "0.1.5";
 pyproject = true;

 disabled = pythonOlder "3.10";

 src = fetchFromGitHub {
 owner = "jnsgruk";
 repo = "pytouchlinesl";
 rev = "refs/tags/${version}";
 hash = "sha256-kdLMuxA1Ig85mH7s9rlmVjEsItXxRlDA1JTFasnJogg=";
 };

 build-system = [ setuptools ];

 dependencies = [
 aiohttp
 pydantic
 ];

 nativeCheckInputs = [
 pytestCheckHook
 pytest-asyncio
 ];

 pythonImportsCheck = [ "pytouchlinesl" ];

 # ...
}

Summary #

And that concludes the first part of this series! Hopefully you found this useful - I’ve learned a lot from people over the years by understanding how they approach problems, so I thought I’d post my methodology here in case it helps anyone refine their process.

I’m certainly no Python expert, so if you’ve spotted a mistake or you think I’m wrong, get in touch.

In the next post, I’ll cover writing the Home Assistant integration, and contributing it to Home Assistant.

Libations: Tailscale on the Rocks

Wed, 21 Aug 2024 00:00:00 +0000

Introduction #

I’m a long-time, self-professed connoisseur of cocktails. I’ve always enjoyed making (and drinking!) the classics, but I also like experimenting with new base spirits, techniques, bitters etc.

Over the years, I’ve collected recipes from a variety of sources. Some originated from books (such as Cocktail Codex and Cocktails Made Easy), others from websites (Difford’s Guide), and most importantly those that I’ve either guessed from things I’ve drunk elsewhere (like my favourite bar in Bristol, The Milk Thistle), or modifications to recipes from the referenced sources.

I wanted somewhere to store all these variations: somewhere that I could search easily from my iPhone (which is nearly always close to me when I’m making drinks). I wanted each recipe to fit in its entirety on my iPhone screen without the need to scroll.

Around the time I started thinking about this problem, I also learned about tsnet and was desperate for an excuse to try it out - and thus Libations was born as the product of two things I love: Tailscale and cocktails!

tswhat? #

Some time ago, Tailscale released a Go library named tsnet. To quote the website:

tsnet is a library that lets you embed Tailscale inside of a Go program

In this case, the embedded Tailscale works slightly different to how tailscaled works (by default, anyway…). Rather than using the universal TUN/TAP driver in the Linux kernel, tsnet instead uses a userspace TCP/IP networking stack, which enables the process embedding it to make direct connections to other devices on your tailnet as if it were “just another machine”. This makes it easy to embed, and drops the requirement for the process to be privileged enough to access /dev/tun.

One of the things I like about how tsnet presents applications as devices on the tailnet, is that you can employ ACLs to control who and what on your tailnet can access the application, rather than the device. I’ve solved this problem before by putting applications in their own systemd-nspawn container and joining those containers to my tailnet. Another nice option is tsnsrv which essentially acts as a Tailscale-aware proxy for individual applications, but in this case I wanted to bake it into the application - which I would only access over my tailnet.

Getting started with tsnet couldn’t be easier:

1
2
3


mkdir tsnet-app; cd tsnet-app
go mod init tsnet-app
go get tailscale.com/tsnet

That will get you set up with a basic Go project, with the tsnet library available. Create a new main.go file with the following contents:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


package main

import (
 "fmt"
 "log"
 "net/http"

 "tailscale.com/tsnet"
)

func main() {
 // Create a new tsnet server instance
 s := tsnet.Server{Hostname: "tsnet-test"}
 defer s.Close()

 // Have the tsnet server listen on :8080
 ln, err := s.Listen("tcp", ":8080")
 if err != nil {
 log.Fatal(err)
 }
 defer ln.Close()

 // Define a very simple handler with a simple Hello, World style message
 handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 fmt.Fprintf(w, "<html><body><h1>Hello from %s, tailnet!</h1>\n", s.Hostname)
 })

 // Start an HTTP server on the tsnet listener
 err = http.Serve(ln, handler)
 if err != nil {
 log.Fatal(err)
 }
}

This is about the most minimal example I could contrive. The code creates a simple instance of tsnet.Server with the hostname tsnet-app, listens on port 8080 and serves up a simple Hello, World! style message. On running the application you’ll see the following:

1
2
3
4
5


❯ go run .
2024/08/13 15:03:37 tsnet running state path /home/jon/.config/tsnet-tsnet-app/tailscaled.state
2024/08/13 15:03:37 tsnet starting with hostname "tsnet-test", varRoot "/home/jon/.config/tsnet-tsnet-app"
2024/08/13 15:03:38 LocalBackend state is NeedsLogin; running StartLoginInteractive...
2024/08/13 15:03:43 To start this tsnet server, restart with TS_AUTHKEY set, or go to: https://login.tailscale.com/a/deadbeeffeebdaed

Clicking the link will open a page in your browser that runs you through Tailscale’s authentication flow, after which you should be able to curl the page directly from any of your devices (assuming you’re not doing anything complicated with ACLs that might prevent it)!

1
2
3
4
5
6
7
8


❯ tailscale status
# ...
100.93.165.28 kara jnsgruk@ linux -
100.106.82.10 tsnet-test jnsgruk@ linux -
# ...

❯ curl http://tsnet-test:8080
<html><body><h1>Hello from tsnet-test, tailnet!</h1>

The library has a pretty small API surface, all of which is documented on pkg.go.dev.

Libations #

For my cocktail app, I wanted to employ a similar, albeit simplified, set of techniques that I use to build this blog. I love Go’s ability to embed static files that can be served as web assets.

Recipe Schema #

I wanted to represent recipes in a format that could be updated by hand if necessary, and easily parsed into a web frontend. I decided to use JSON for this, representing the recipes as a list:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


[
 {
 "id": 10,
 "name": "New York Sour",
 "base": ["Bourbon"],
 "glass": ["12oz Lowball"],
 "method": ["Dry Shake", "Shake"],
 "ice": ["Cubed"],
 "ingredients": [
 { "name": "Lemon Juice", "measure": "20", "unit": "ml" },
 { "name": "Sugar", "measure": "20", "unit": "ml" },
 { "name": "Red Wine", "measure": "10", "unit": "ml" },
 { "name": "Bourbon", "measure": "40", "unit": "ml" },
 { "name": "Egg White", "measure": "20", "unit": "ml" }
 ],
 "garnish": ["Lemon Sail"],
 "notes": "Use claret or malbec"
 }
]

This schema is able to capture all the relevant details from the different formats I’ve seen over the years. It would take some time to format my favourites into this schema, but that was always going to be the case.

I was fortunate enough to get access to the recipe collection from a well regarded cocktail bar in the UK. Unfortunately it was given to me in a hard-to-parse PDF, which resulted in many hours of playing with OCR tools and manual data cleaning - but enabled me to bootstrap the app with around 450 high quality recipes. I didn’t include their recipes in the libations repository, but I did include some of my own favourite concoctions in a sample recipe file. My Mezcal Margarita gets pretty good reviews 😉.

Server #

The server implementation needed to fulfil a few requirements:

Parse a specific recipes file, optionally passed via the command line
Have an embedded filesystem to contain static assets and templates
Be able to render HTML templates with the given recipes
Listen on either a tailnet (via tsnet), or locally (for testing convenience)
When listening on the tailnet, listen on HTTPS, redirecting HTTP traffic accordingly

I wanted to keep dependencies to a minimum to make things easier to maintain over time. The tsnet library pulls in a few indirect dependencies, but everything else Libations uses is in the Go standard library.

The recipes JSON schema is very simple, and is modelled with a couple of Go structs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


// Ingredient represents the name and quantity of a given ingredient in a recipe.
type Ingredient struct {
 Name string
 Measure string
 Unit string
}

// Drink represents all of the details for a given drink.
type Drink struct {
 ID int
 Name string
 Base []string
 Glass []string
 Method []string
 Ice []string
 Ingredients []Ingredient
 Garnish []string
 Notes string
}

The parseRecipes function checks whether or not the user passed the path to a specific recipe file, or whether it should default to parsing the sample recipes file. Once it’s determined the right file to parse, and validated its existence, it unmarshals the JSON using the Go standard library.

Users have the option of passing the -local flag when starting Libations, which bypasses tsnet completely and starts a local HTTP listener on the specific port. This makes for easier testing when iterating through changes to the web UI and other elements:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


// ...
addr := flag.String("addr", ":8080", "the address to listen on in the case of a local listener")
local := flag.Bool("local", false, "start on local addr; don't attach to a tailnet")

// ...
var listener *net.Listener
if *local {
 listener, err = localListener(*addr)
} else {
 listener, err = tailscaleListener(*hostname, *tsnetLogs)
}
if err != nil {
 slog.Error("failed to create listener", "error", err.Error())
 os.Exit(1)
}
//...

Setting up the tsnet server and listener is only mildly more complicated – but mostly due to my requirement that all HTTP traffic is redirected to HTTPS, using the LetsEncrypt certificates that Tailscale provides automatically. The redirects are handled by a separate Goroutine in this case:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


// Start a standard HTTP server in the background to redirect HTTP -> HTTPS.
go func() {
 httpLn, err := tsnetServer.Listen("tcp", ":80")
 if err != nil {
 slog.Error("unable to start HTTP listener, redirects from http->https will not work")
 return
 }

 slog.Info(fmt.Sprintf("started HTTP listener with tsnet at %s:80", hostname))

 err = http.Serve(httpLn, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 newURL := fmt.Sprintf("https://%s%s", r.Host, r.RequestURI)
 http.Redirect(w, r, newURL, http.StatusMovedPermanently)
 }))
 if err != nil {
 slog.Error("unable to start http server, redirects from http->https will not work")
 }
}()

With the correct listener selected, I create an http.ServeMux to handle routing to static assets and rendering templates, and pass that mux to the http.Serve method from the Go standard library - and that’s it! At the time of writing the Go code totals 235 lines - not bad!

Web Interface #

The web interface was designed primarily for mobile devices, and I’ve not yet done the work to make it excellent for larger-screened devices - though it’s certainly bearable. It’s also read-only at the moment: you can browse all the recipes, and there is a simple full-text search which can narrow the list of recipes down by searching for an ingredient, method, glass type, notes, etc.

As mentioned in an earlier post, I’m a big fan of the Vanilla Framework, which is a “simple, extensible CSS framework” from Canonical, and is used for all of Canonical’s apps and websites. Given that I had some prior experience using it, I decided to use it again here. I started using my tried and tested recipe of Vanilla + Hugo, but later reverted to using simple HTML templates with Go’s html/template package.

The result is a set of templates, which iterate over the recipe data from the JSON file, and output nicely styled HTML elements:

There is nothing fancy going on here - it’s nearly all stock Vanilla Framework. I do specify some overrides to make the colours a bit less Ubuntu-ish, but that’s it!

One detail I’m pleased with is the dynamic drink icons. The icons indicate the type of glass the particular drink should be served in. This is a simple trick: for each drink the HTML template renders a glass-icon partial, which reads the glass type specified in the recipe, and renders the appropriate SVG file which is then coloured with CSS.

Packaging for NixOS #

There were two main tasks in this category: creating the Nix package itself, and writing a simple NixOS module that would make it simple for me to run it on my NixOS server.

The project uses a Flake to provide the package, overlay and module. The standard library in Nix has good tooling for Go applications now, meaning the derivation is short:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


{
 buildGo122Module,
 lastModifiedDate,
 lib,
 ...
}:

let
 version = builtins.substring 0 8 lastModifiedDate;
in
buildGo122Module {
 pname = "libations";
 inherit version;
 src = lib.cleanSource ../.;
 vendorHash = "sha256-AWvaHyJL7Cm+zCY/vTuTAsgLbVy6WUNfmaGbyQOzMMQ=";
}

I haven’t cut any versioned releases of Libations at the time of writing - I’m using the last modified date of the flake to version the binaries.

The module starts the application with systemd, and optionally provides it with a recipes file. There are four options defined at the time of writing: services.libations.{enable,recipesFile,tailscaleKeyFile,package}:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


options = {
 services.libations = {
 enable = mkEnableOption "Enables the libations service";

 recipesFile = mkOption {
 type = nullOr path;
 default = null;
 example = "/var/lib/libations/recipes.json";
 description = ''
 A file containing drinks recipes per the Libations file format.
 See https://github.com/jnsgruk/libations.
 '';
 };

 tailscaleKeyFile = mkOption {
 type = nullOr path;
 default = null;
 example = "/run/agenix/libations-tsauthkey";
 description = ''
 A file containing a key for Libations to join a Tailscale network.
 See https://tailscale.com/kb/1085/auth-keys/.
 '';
 };

 package = mkPackageOption pkgs "libations" { };
 };
};

The tailscaleKeyFile option enables the service to automatically join a tailnet using an API key, rather than prompting the user to click a link and authorise manually.

These options are translated into a simple systemd unit:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


config = mkIf cfg.enable {
 systemd.services.libations = {
 description = "Libations cocktail recipe viewer";
 wantedBy = [ "multi-user.target" ];
 after = [ "network.target" ];
 environment = {
 "XDG_CONFIG_HOME" = "/var/lib/libations/";
 };
 serviceConfig = {
 DynamicUser = true;
 ExecStart = "${cfg.package}/bin/libations -recipes-file ${cfg.recipesFile}";
 Restart = "always";
 EnvironmentFile = cfg.tailscaleKeyFile;
 StateDirectory = "libations";
 StateDirectoryMode = "0750";
 };
 };
}

The XDG_CONFIG_HOME variable is set so that tsnet stores it’s state in /var/lib/libations, rather than trying to store it in the home directory of the dynamically created user for the systemd unit.

I use agenix on my NixOS machines to manage encrypted secrets, and for this project I used it to encrypt both the initial Tailscale auth key, and my super-secret recipe collection! The configuration to provide the secrets and configure the server to run libations is available on Github, but looks like so:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


{ config, self, ... }:
{
 age.secrets = {
 libations-auth-key = {
 file = "${self}/secrets/thor-libations-tskey.age";
 owner = "root";
 group = "root";
 mode = "400";
 };

 libations-recipes = {
 file = "${self}/secrets/thor-libations-recipes.age";
 owner = "root";
 group = "root";
 mode = "444";
 };
 };

 services.libations = {
 enable = true;
 recipesFile = config.age.secrets.libations-recipes.path;
 tailscaleKeyFile = config.age.secrets.libations-auth-key.path;
 };
}

As a result, the application is now available at https://libations, with a valid LetsEncrypt certificate, on all of my machines! 🎉

Summary #

This was a really fun project. It felt like a fun way to explore tsnet, and resulted in something that I’ve used a lot over the past year. I don’t have many plans to adjust things in the near future - though I do find myself wanting a nice interface to add new recipes from time to time.

And now for the twist: three weeks ago I gave up drinking alcohol (likely for good), so now I’m on a mission to find some non-alcoholic recipes to make life a little tastier! I suspect this will be hard work - and I’ll certainly miss some of my favourites, but my wife and I have already found some compelling alternatives.

If you’ve got a favourite recipe (alcoholic or not) and you liked the article, then perhaps open a PR and add it to the sample recipes file!

Secure Boot & TPM-backed Full Disk Encryption on NixOS

Sat, 20 Apr 2024 00:00:00 +0000

Update: Since I wrote this post, I have been made aware of a particular situation where, at the time I write this (2025-01-17), the steps described in this article will result in a setup that is still (in many cases) vulnerable to an attack where the attacker has physical access to the machine. This may be acceptable in your threat model, but I’d encourage you to read the excellent article to gain a full understanding of the issue.

Introduction #

For the last decade (whoa…) or so, I’ve defaulted to using LUKS-encrypted drives for my machines. In general, I configure an unencrypted boot/EFI partition, then place either an ext4 or btrfs filesystem inside a LUKS container which is used for the root partition.

Some of my machines also have extra disks: my desktop has a 1TB NVMe drive for root, and a 2TB NVMe “data” drive mounted in my home directory under /home/jon/data. I don’t like having to type two different encryption passphrases at boot, so I usually have the extra disk automatically unlocked by putting the key in a file on the root drive, and placing an entry in /etc/crypttab.

This setup works fine for desktop machines, but it’s cumbersome on headless machines because unattended reboots require the disk passphrase to be entered at boot. Even then, all of my computers are exclusively used by me, and this setup means I have to enter two passwords on every boot to get to a working desktop environment (one for the disk, and one for the login manager).

I solved this recently with a combination of Secure Boot and a Trusted Platform Module (TPM), so let’s look at those first with a brief and high-level overview of each.

What’s a TPM? #

Most machines that have been manufactured in the last decade, and certainly in the last 5 years, contain a cryptographic coprocessor conforming to the Trusted Platform Module (TPM) spec. The TPM is a dedicated microcontroller primarily used for verifying the integrity of a machine.

TPMs can be used for storing cryptographic key material and performing basic cryptographic operations. The general premise is that keys can be loaded into the TPM, which enables the TPM to perform cryptographic operations using that key (signing, encrypting, etc.), but the key cannot be recovered or read from the TPM unless certain conditions are met. TPMs also provide other facilities such as secure random number generation, which in turn enables them to securely generate cryptographic keys.

Verifying system integrity essentially boils down to being able to ensure the machine hasn’t been tampered with between boots, and that the boot process itself hasn’t been compromised. A given firmware or operating system can take hardware “measurements” and store those measurements in dedicated slots called Platform Configuration Registers (PCRs). The measurements pertain to the underlying hardware and configuration of the machine. The TPM itself never performs the actual verification of the PCRs, and in fact has no knowledge of whether a measurement is inherently “good” or “bad”, but it can provide signed attestations of their values, which are then judged by the application requesting the attestation according to some policy.

Each PCR contains a hash representing a particular hardware measurement, which can be read at any time, but cannot be overwritten. Rather than allowing a traditional write operation, PCRs are updated through an “extend” operation which depends on the previous hash value, creating a chain of trust not dissimilar from how a blockchain is formed. This means that a given measurement can never be fully removed from the TPM.

Once the measurements are stored in the PCRs, there are various times and purposes for which the firmware or an operating system might read them - one example is for remote attestation during login to a system. In this scenario the attestation can be used to verify that the machine hasn’t been tampered with (perhaps in an Evil Maid attack). One could also store a Certificate Authority signing key in a TPM, and have the Certificate Authority software interface with the TPM to sign certificates using the PKCS#11 standard.

My use-case is to enabling the TPM to provide the passphrase to unlock a LUKS-encrypted disk, which is what I’ll focus on in this post.

Secure Boot #

Secure Boot is the mechanism by which the code executed by a machine’s Unified Extensible Firmware Interface (UEFI) can be verified as trusted. In the vast majority of cases, the first thing executed by the UEFI is a bootloader.

When Secure Boot is enabled, each binary executed by the UEFI must contain a checksum and a signature - which the UEFI verifies before launching the code. In the case that either the checksum or signature do not match, the UEFI will refuse the execute the code, and the boot process will halt.

Many OEM machines ship with Microsoft Windows installed, and thus ship with the necessary keys to validate signatures created with Microsoft’s certificate authority. Linux systems are able to utilise these keys through shim - a small and easily verifiable piece of software which is signed by Microsoft. shim sits between the UEFI and the bootloader in the boot process, obviating the need for every Linux bootloader to be signed by Microsoft on every release. The shim is designed to extend trust from the keys trusted by the computer’s firmware to a new set of keys controlled by the operating system.

But what does Secure Boot get us in reality? By signing the kernel, and in some cases a single UEFI PE binary known as a Unified Kernel Image (UKI) (which contains the bootloader, the kernel, the command-line used to boot the kernel, and other resources), one can be reasonably sure that the boot process hasn’t been tampered with.

This process thwarts a number of common physical attack vectors, such as manipulating the kernel command line to bypass the machine’s login and drop straight to a root shell - and combined with disk encryption can prevent offline data transfer from the machine. It also defends against malware which compromises the operating system’s boot process such that it can start before the OS and obfuscate it’s presence.

Threat Modelling #

As with any security measure, Secure Boot is not a silver bullet. You should always consider your own personal threat model, and the sorts of attacks you’re looking to defend against.

For example, Secure Boot can help prevent the sort of malware infections described in the previous section, but if an attacker gets physical access to your machine, and the UEFI isn’t adequately protected, they could simply disable secure boot and carry on unhindered.

I mitigate this by password protecting the UEFI. This isn’t perfect, but is likely sufficient protection for my threat model which is more about protecting chancers and petty thieves from gaining access to my information, than from determined attackers who gain physical access to my property.

Storing keys in a TPM is theoretically safe, in that each TPM has a unique seed which cannot be retrieved, and enables the TPM to deterministically generate keys between reboots. It’s very difficult to retrieve the seed, and thus very difficult to duplicate a TPM, but not impossible. Even then, the Linux kernel’s communication with the TPM on-the-wire is unencrypted, and the same can be said for many other subsystems which use the TPM. A recent example of this vulnerability was demonstrated by sniffing a Bitlocker key off the LPC bus in a Lenovo X1 Carbon laptop (using a Raspberry Pi Pico, no less). In many modern machines this is mitigated by the TPM being on-CPU, but the point still stands.

I choose to enroll Microsoft’s platform keys, which in theory degrades the security of my device in the case that Microsoft’s signing key is compromised, though all of my machines are compatible with fwupd and can receive updates to the database through that mechanism if required (and in fact have done in the past 18 months). This could be further mitigated by using custom keys and certificates for the full chain, but this is more overhead for daily operations and updates. It’s also worth considering whether you have the resources to fully secure your own chain - especially by comparison to Microsoft who spend tens of millions of dollars per year on security. If an attacker wants your information, and are able to compromise Microsoft’s CA, your own CA may not be such a hurdle.

Security measures are always a trade-off between Confidentiality, Availability and Integrity (CIA). In general, the more rigidly secure boot is implemented and configured, the more you’re protecting confidentiality and integrity. The choices I’ve made are slightly more in favour of availability, but nonetheless raise the bar for any attacker significantly.

Enabling Secure Boot on NixOS #

Now for the fun part! The process for enabling Secure Boot on NixOS has simplified in recent months owing to the creation of lanzaboote - a project which takes of preparing and signing Unified Kernel Images containing a custom stub, the bootloader, the Linux kernel, the kernel’s initrd and the kernel command line. lanzaboote also takes care of installing the UKI on the ESP partition so the UEFI can execute it at boot.

The lanzaboote stub differs slightly from systemd-stub, in that it doesn’t require the kernel and initrd to be part of the UKI. This is important for a generation-based operating system like NixOS because bundling the kernel and initrd into a new UKI for every generation would consume a lot of disk space, and quickly exhaust the ESP on most machines. In lanzaboote’s implementation, the kernel and initrd are stored separately on the ESP, and the chain of trust is preserved by validating the signature of the kernel, and embedding a cryptographic hash of the initrd into the signed UKI.

The project takes advantage of systems that have bootspec enabled, which is a relatively recent NixOS RFC that ensures configured machines maintain a file containing a set of memoised facts about a system’s closure. Bootspec aims to “provide more uniform feature support” to bootloaders in the NixOS ecosystem and “enable NixOS users to implement custom bootloader tools and policy” - of which lanzaboote is one.

Generating Secure Boot Keys #

The first step is to generate some keys for the secure boot process. This can be achieved using the sbctl package:

1
2
3
4


❯ sudo nix run nixpkgs#sbctl create-keys
Created Owner UUID 6ac34cc3-a23d-9745-ef33-a03f523d20a3
Creating secure boot keys...✓
Secure boot keys created!

This should only take a few seconds at maximum, and will result in a set of keys being populated in /etc/secureboot:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


❯ tree /etc/secureboot
/etc/secureboot
├── files.db
├── GUID
└── keys
 ├── db
 │ ├── db.key
 │ └── db.pem
 ├── dbx
 │ ├── dbx.key
 │ └── dbx.pem
 ├── KEK
 │ ├── KEK.key
 │ └── KEK.pem
 └── PK
 ├── PK.key
 └── PK.pem

Enable Bootspec #

Ensure that bootspec is enabled in your Nix configuration. You can see this in my flake for my desktop machine on Github:

1
2
3


{
 boot.bootspec.enabled = true;
}

Enable `lanzaboote` #

I use a flake to configure all of my machines, so I’m able to get access to lanzaboote by adding the upstream flake as an input to my own nixos-config flake:

1
2
3
4
5


# ...
inputs = {
 lanzaboote.url = "github:nix-community/lanzaboote";
};
# ...

Once you’ve added lanzaboote as a dependency, you’ll need to import the lanzaboote module:

1
2
3


# ...
imports = [ lanzaboote.nixosModules.lanzaboote ];
# ...

In my flake, I use a custom helper function to build NixOS configurations, so the module is passed directly to lib.nixosSystem through the modules attribute.

The lanzaboote module replaces the systemd-boot module, and as such you must explicitly disable systemd-boot when enabling lanzaboote. Additionally, if you wish to use the TPM for disk unlock (described in the next section), you must use the systemd initrd hooks (or something like clevis):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


boot = {
 initrd.systemd.enable = true;

 loader.systemd-boot.enable = lib.mkForce false;

 lanzaboote = {
 enable = true;
 pkiBundle = "/etc/secureboot";
 };
};

This is represented in my config here.

Once enabled, rebuild your system (in my case with sudo nixos-rebuild switch --flake /home/jon/nixos-config) and verify that your machine is ready for Secure Boot. Don’t panic about the kernel images being reported as not signed, this is expected:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


❯ sudo nix run unstable#sbctl verify
Verifying file database and EFI images in /boot...
✓ /boot/EFI/BOOT/BOOTX64.EFI is signed
✓ /boot/EFI/Linux/nixos-generation-414-376jna572gsb23snqs67t7s4bwxzb3epblmdnzweghuepopml2va.efi is signed
✓ /boot/EFI/Linux/nixos-generation-415-iqulgohymbdppgtxzho6ou3fcuxjbxhumpzm4vojmipwy3sbmuna.efi is signed
✓ /boot/EFI/Linux/nixos-generation-416-kxnzioafnduwwck3oypo7rqwtoat745czp2bpehoufp4yqiawypa.efi is signed
✗ /boot/EFI/nixos/kernel-6.8.2-242idodyvf36cpl6s5dskjy6mo4tjhszuwa3hye7qcjyuo5vnehq.efi is not signed
✗ /boot/EFI/nixos/kernel-6.8.5-zqulrwsucm6okcyns6v2jhh6fregk3bvsdth3yloqfymfbgnh64a.efi is not signed
✗ /boot/EFI/nixos/kernel-6.8.7-6mmixkr6ewywm5swgbi5ethbpgnyia4borzmkevcjx7n7t3mtida.efi is not signed
✓ /boot/EFI/systemd/systemd-bootx64.efi is signed

Prepare the UEFI #

Reboot your machine and enter the UEFI interface. This part of the process will vary from machine to machine depending on the UEFI implementation, but you’re looking to enable Secure Boot, and clear the preloaded Secure Boot keys. This may be referred to as “Setup Mode”, or erasing the “Platform Keys”.

While you’re here, I’d also advise setting a UEFI password before rebooting back into NixOS.

Enroll Secure Boot Keys #

The final stage in the process is to enroll your newly generated Secure Boot keys from step 1 into the UEFI. This is again achieved with sbctl:

1
2
3
4


❯ sudo nix run nixpkgs#sbctl enroll-keys -- --microsoft
Enrolling keys to EFI variables...
With vendor keys from microsoft...✓
Enrolled keys to the EFI variables!

I chose to use the --microsoft option to also enroll the UEFI vendor certificates from Microsoft. Some systems contain firmware that is signed and validated when Secure Boot is enabled, and omitting the Microsoft keys could prevent your device from booting - omit this option with caution!

Verify Secure Boot #

Once you’ve enrolled the keys, reboot the machine back into NixOS and use bootctl to confirm that Secure Boot is in fact enabled:

1
2
3
4
5
6
7


❯ bootctl status
System:
 Firmware: UEFI 2.80 (American Megatrends 5.26)
 Firmware Arch: x64
 Secure Boot: enabled (user)
 TPM2 Support: yes
 Boot into FW: supported

TPM Unlock of Root Partition #

Now that we’re (reasonably) confident that no one can tamper with the boot process, we can progress to allowing the machine to auto-unlock the encrypted disk using a key stored in the TPM.

This is actually more common than you might think - Windows has enabled this behaviour by default for some time with Bitlocker disk encryption, and Canonical is also working on bringing TPM-backed full disk encryption to Ubuntu.

This is probably the easiest step of them all! A simple invocation of systemd-cryptenroll is all that’s required. The arguments below instruct the machine that PCRs 0, 2, 7 and 12 should be measured and verified before the TPM is allowed to unlock the disk.

According to the Linux TPM PCR Regsitry, this means the following are measured before the LUKS key is presented:

PCR 0: Core system firmware executable code
PCR 2: Extended or pluggable executable code
PCR 7: SecureBoot state
PCR 12: Kernel command line, system credentials and system configuration images

1

❯ sudo systemd-cryptenroll --tpm2-device=auto --tpm2-pcrs=0+2+7+12 --wipe-slot=tpm2 /dev/nvme0n1p2

And that’s it! The next time you reboot, your disk should be automatically unlocked by the TPM, and your machine should boot straight to your display manager, or the TTY login if no display manager is configured.

Useful Resources #

None of the knowledge in this post is novel, but rather the culmination of some knowledge acquired over the past few years, and some more targeted reading more recently. In the process, I learned a bunch from the following:

Summary #

About 5 years ago, I was sporting a fully secure-boot enabled Dell XPS 13 running Arch Linux. Back then, the process was complicated, manual, and required a lot of maintenance between upgrades. For me, it was more pain than gain, but an interesting learning experience nonetheless.

When I sat down earlier this year to enable Secure Boot on NixOS, I’d set aside a few hours. I was astounded that 10 minutes later I was finished. I wrote this post as a memo to my future self, but also to illustrate how simple it can be to enable Secure Boot and TPM disk unlock in 2024.

I don’t claim to be an expert on the inner workings of TPMs, nor Secure Boot. The things I can say for certain are that TPMs are complex, that there are improvements that could be made to Linux’s interactions with the TPM, and that a determined and well-resourced attacker is likely going to succeed one way or another.

If you spot an inaccuracy in this post, reach out and let me know on Mastodon, on Telegram, by email, or however you prefer!

Until next time!

Update 2024/04/29: Thanks to @pimeys for pointing out that one must enable the systemd initrd hooks systemd-cryptenroll to function correctly, and also that PCR 12 must be measured to prevent the LUKS key from being released if the kernel command line has been modified.

A homelab dashboard for NixOS

Tue, 05 Mar 2024 00:00:00 +0000

Introduction #

I run a very small homelab that provides some basic services to my home network. I’m not much of a data hoarder, but my lab consists of some redundant storage in a raidz2 ZFS pool, and I use the homelab as a receive-only target for Syncthing, and as the point from which backups of my critical data are made using Borg & Borgbase.

It also runs a few other small services - all of which are exclusively available over Tailscale to my other devices. I wanted a small dashboard solution that could give me links to each of those services with a nice simple URL.

There are certainly plenty of options; this seems to be a highly crowded space in the open source homelab world. I settled on the rather ambiguously named homepage. At the time of writing, my dashboard looks like so, though there are people who have been far more creative with the appearance!

Naturally, I wanted to run this on NixOS, so in July 2023 I landed one of my early contributions to the project in the form of PR #243094 which added the package (named homepage-dashboard), a basic NixOS module and a basic test.

Homepage’s Configuration #

Homepage is configured using a set of YAML files named services.yaml, bookmarks.yaml, widgets.yaml, etc. When I originally started writing the module, it would look for those config files in a hard-coded location (/config), and if the files were missing it would copy a skeleton config into place with some defaults to get you going.

The hard-coded location results from the fact that the upstream primarily support deploying Homepage using Docker. They expect the config directory to be bind-mounted into the container from the host. As part of the initial packaging effort, I contributed a patch upstream to allow customising this location by setting the HOMEPAGE_CONFIG_DIR environment variable, which I then set in the systemd unit configuration in the NixOS module to /var/lib/homepage-dashboard.

This has been working fine for a few months, but it’s been bugging me that my dashboard configuration is not part of the declarative system configuration. Once the initial skeleton had been copied in place, you were left to edit the files manually (and back them up) if you wanted to make changes. Moreover, homepage defaults to creating a logs directory as a subdirectory of the config directory. This makes some sense in a container environment, but given that homepage also logs to stdout (which is collected by the systemd journal on NixOS), it’s really just unnecessary duplication.

Evolving The Module Design #

Before writing the actual implementation of the module, I decided to first sketch out what I wanted my NixOS configuration to look like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


{
 services.homepage-dashboard = {
 # These options were already present in my configuration.
 enable = true;
 package = unstable.homepage-dashboard;

 # The following options were what I planned to add.

 # https://gethomepage.dev/latest/configs/settings/
 settings = {};

 # https://gethomepage.dev/latest/configs/bookmarks/
 bookmarks = [];

 # https://gethomepage.dev/latest/configs/services/
 services = [];

 # https://gethomepage.dev/latest/configs/service-widgets/
 widgets = [];

 # https://gethomepage.dev/latest/configs/kubernetes/
 kubernetes = { };

 # https://gethomepage.dev/latest/configs/docker/
 docker = { };

 # https://gethomepage.dev/latest/configs/custom-css-js/
 customJS = "";
 customCSS = "";
 };
}

Each of the new sections would then map neatly to the different configuration section in the upstream documentation. Based on my learning from the Scrutiny module, I wanted to utilise the same RFC42 approach which would obviate the need for the module to specify every possible supported configuration option, resulting in a large, difficult to maintain module which could quickly fall behind the upstream project.

Homepage supports a large number of widgets (see below) which are able to scrape information from the API of various devices and services. These often require an API key or token of some kind, and having those in plaintext as part of the machine configuration is undesirable from a security perspective - even if your services are all on a private network like mine. Luckily I found out (tucked away in the docs) that Homepage can inject secret values into the configuration using environment variables.

Given the template value of {{HOMEPAGE_VAR_FOOBAR}} as part of the configuration, Homepage will automatically substitute the value of the variable HOMEPAGE_VAR_FOOBAR.

I decided to provide a single configuration option named environmentFile so that users can supply the path to an environment file containing all of their variables. This file can be omitted from Git repositories and configurations, or included in encrypted form. I achieve this by including the file encrypted using agenix which integrates @FiloSottile’s wonderful age into NixOS. You can see how that’s supplied as part of my nixos-config.

Backwards Compatibility #

According to my relatively naive Github search I estimated that there are not that many users of the module - likely in the tens, rather than the hundreds or thousands. That said, I think its important not to break those users. There’s no reason to expect that a nix flake update should break your system.

The way I chose to handle this in the module was to check if any of the new config options are set. If they’re not, the module behaves as before, but displays a deprecation warning:

The implementation of this check is relatively crude, but it works, and it will only be around until the release of NixOS 24.05 (in May 24):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


config =
 let
 # If homepage-dashboard is enabled, but none of the configuration values have been updated,
 # then default to "unmanaged" configuration which is manually updated in
 # var/lib/homepage-dashboard. This is to maintain backwards compatibility, and should be
 # deprecated in a future release.
 managedConfig = !(
 cfg.bookmarks == [ ] &&
 cfg.customCSS == "" &&
 cfg.customJS == "" &&
 cfg.docker == { } &&
 cfg.kubernetes == { } &&
 cfg.services == [ ] &&
 cfg.settings == { } &&
 cfg.widgets == [ ]
 );

 configDir = if managedConfig then "/etc/homepage-dashboard" else "/var/lib/homepage-dashboard";

 msg = "using unmanaged configuration for homepage-dashboard is deprecated and will be removed"
 + " in 24.05. please see the NixOS documentation for `services.homepage-dashboard' and add"
 + " your bookmarks, services, widgets, and other configuration using the options provided.";
 in
 lib.mkIf cfg.enable {
 # Display the deprecation warning if the configuration isn't managed
 warnings = lib.optional (!managedConfig) msg;

# ...

Solving Log Duplication #

I mentioned in a previous section that Homepage logs to both stdout and a logs directory by default. While the log file path can be customised, it’s not currently possible to disable the file logging completely. It’s not desirable to have the log file in this context, because all of the logs are collected by systemd anyway.

Looking at the upstream implementation, the logger is instantiated and configured in a single logger.js file. Homepage has a policy that they won’t accept feature contributions (even if you do the implementation) unless the feature gets at least 10 upvotes. I filed a feature request, but it’s yet to get enough votes to be accepted.

In the mean time I wrote a short patch on a branch in my personal fork which makes Homepage adhere to an environment variable named LOG_TARGETS. The possible values are both, file or stdout with a default value of both to respect the existing behaviour and remain backward compatible. The patch is now applied in the Nix package as part of nixpkgs, and the module configures the systemd unit by setting the LOG_TARGETS variable to stdout in cases where the configuration is managed:

1
2
3
4
5
6
7
8
9


{
 # ...
 environment = {
 HOMEPAGE_CONFIG_DIR = configDir;
 PORT = toString cfg.listenPort;
 LOG_TARGETS = lib.mkIf managedConfig "stdout";
 };
 # ...
}

Update: My feature request got upvoted pretty quickly, and as a result my pull request was merged. This means that following the next release of Homepage, I’ll be able to drop the patch from the Nix package. Thanks to all those who upvoted it!

Bolstering The Test Suite #

When I originally implemented the tests for the module, they simply enabled the service and ensure that it responded on the specified port. I wanted to include some logic in the test that ensured the ability to detect when managed configuration should be used, and when the module should respect an existing implementation.

The NixOS test suite supports specifying multiple machines as part of a given test, so extending the previous implementation wasn’t particularly cumbersome. See below for the (annotated) implementation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49


import ./make-test-python.nix ({ lib, ... }: {
 name = "homepage-dashboard";
 meta.maintainers = with lib.maintainers; [ jnsgruk ];

 # Create a machine that uses the legacy module format,
 # where configuration is unmanaged by nix, and relies
 # upon YAML files.
 nodes.unmanaged_conf = { pkgs, ... }: {
 services.homepage-dashboard.enable = true;
 };

 # Create another machine that sets some simple
 # configuration using the new module system. This
 # doesn't need to be exhaustive, just enough to trigger
 # the condition that makes the module use managed config.
 nodes.managed_conf = { pkgs, ... }: {
 services.homepage-dashboard = {
 enable = true;
 settings.title = "custom";
 };
 };

 testScript = ''
 # Ensure the services are started on unmanaged machine,
 # and that the service responds to HTTP requests on the
 # expected port.
 unmanaged_conf.wait_for_unit("homepage-dashboard.service")
 unmanaged_conf.wait_for_open_port(8082)
 unmanaged_conf.succeed("curl --fail http://localhost:8082/")

 # Ensure that /etc/homepage-dashboard doesn't exist, and boilerplate
 # configs are copied into place in `/var/lib/homepage-dashboard`.
 # This validates the existing behaviour.
 unmanaged_conf.fail("test -d /etc/homepage-dashboard")
 unmanaged_conf.succeed("test -f /var/lib/private/homepage-dashboard/settings.yaml")

 # Ensure the services are started on managed machine,
 # and that the service responds to HTTP requests on the
 # expected port.
 managed_conf.wait_for_unit("homepage-dashboard.service")
 managed_conf.wait_for_open_port(8082)
 managed_conf.succeed("curl --fail http://localhost:8082/")

 # Ensure /etc/homepage-dashboard is created and unmanaged
 # conf location isn't present
 managed_conf.succeed("test -d /etc/homepage-dashboard")
 managed_conf.fail("test -f /var/lib/private/homepage-dashboard/settings.yaml")
 '';
})

This is by no means exhaustive, and I can certainly imagine increasing the coverage here at a later date, but it does at least give some confidence when working on the module that the two basic modes of operation are functioning correctly.

Migrating Existing Configurations #

If you have been using the module in its past form, you may be wondering what the easiest way to migrate to the new format is…

I made the shift using yaml2nix to convert my existing YAML configurations to Nix expressions, and then formatted the output using nixpkgs-fmt. For example, given the following settings.yaml (which came from my homelab before I moved over):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


---
# For configuration options and examples, please see:
# https://gethomepage.dev/en/configs/settings

title: sgrs dashboard
favicon: https://jnsgr.uk/favicon.ico
headerStyle: clean

layout:
 media:
 style: row
 columns: 3
 infra:
 style: row
 columns: 4
 machines:
 style: row
 columns: 4

You can do the following to get a Nix expression that can be assigned to services.homepage-dashboard.settings in your machine configuration, converting the YAML to a Nix expression:

 ~/temp
❯ nix run nixpkgs#yaml2nix settings.yaml
{ title = "sgrs dashboard"; favicon = "https://jnsgr.uk/favicon.ico"; headerStyle = "clean"; layout = { media = { style = "row"; columns = 3; }; infra = { style = "row"; columns = 4; }; machines = { style = "row"; columns = 4; }; }; }

With that output, you can insert a few line breaks and rely on nixpkgs-fmt to get everything lined up properly. You can see my complete dashboard configuration in Nix format as part of my nixos-config repository.

Summary #

The PR was merged earlier today, and will now need to trickle through the branches on its way to nixos-unstable. At the time of writing, it hasn’t quite made it there:

You can track for yourself on the nixpkgs tracker, but the time delay should give you a chance to migrate your configuration!

See Github for the full module implementation, package and tests.

Let me know if you’re using the module, or if you run into any issues! If you’re a fan of Homepage, then consider helping out with the project or sponsoring them on Github, and once again thank you to those who helped review and shape the module as part of this upgrade!

Contributing Scrutiny to nixpkgs

Mon, 19 Feb 2024 00:00:00 +0000

Introduction #

I’m quite overwhelmed by the number of people who read, and engaged with my last post on packaging Scrutiny for NixOS. I hadn’t intended on posting to this blog more than maybe once per month, but I decided to continue documenting the journey of landing Scrutiny upstream in nixpkgs.

One of the things that really struck me about NixOS early on was how simple the contribution process is. A single Github repository contains all the packages, all the modules, and all the tests.

I’ve found the community incredibly helpful and respectful in all of my contributions - often with 2-3 people quickly providing reviews to help shape the contribution. I’ve heard complaints that reviews can be quite pedantic, and while I agree to some extent, the reviews nearly always come with concrete suggestions which are a great way to learn. Notwithstanding the fact that you are contributing to an operating system package that could land on many thousands of machines - it’s important to be careful!

If you’ve been deliberating about whether or not to submit your package, module or otherwise, I’d encourage you to go for it. I’ve always found it rewarding and it’s taught me a lot about Nix and NixOS.

My first contribution was adding a package, module and test for multipass - as you can see I had plenty to learn, but people were generous with their time and it resulted in a contribution that I use daily to get my work done on NixOS.

For those losing sleep over the poorly disk I showed in my last post, you’ll be relieved to know that it’s now been replaced! Incidentally this was the first time I’ve replaced a disk in a ZFS array and I was super impressed with how easy the process was!

Improving My Work #

Over the weekend following my first post about Scrutiny, a couple of people identified some small issues and made pull requests to resolve them - thank you to those people! You can see their work here, here and here. Someone also pointed out to me that string-interpolating YAML wasn’t the most ideal solution to rendering the configuration file, and the I could have used builtins.toJSON, taking advantage of the fact that JSON is also valid YAML.

I also wanted to make sure that there was a way for people to use the module with an existing InfluxDB deployment. My original implementation assumed that it could just add services.influxdb2.enable = true to the system configuration and use the default credentials to connect. I added some options to the module for configuring a different InfluxDB instance if required. These were later modified slightly to use lib.types.nullOr lib.types.str following a helpful review comment.

Next up was refactoring the helper function I wrote for generating the configuration file. As mentioned above, Nix comes with a handy builtins.toJSON function which takes a native Nix expression (an attribute set in this case) and renders it to JSON. This obviates the need to string-interpolate YAML, which is error prone at the best of times. The renewed function looked like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


{
 mkScrutinyCfg = cfg: pkgs.writeTextFile {
 name = "scrutiny.yaml";
 text = builtins.toJSON {
 version = 1;
 web = {
 listen = {
 inherit (cfg) host port;
 basepath = if cfg.basepath != "" then "/" + cfg.basepath else "";
 };

 database = {
 location = "/var/lib/scrutiny/scrutiny.db";
 };

 src = {
 frontend.path = "${pkgs.scrutiny}/share/scrutiny";
 };

 influxdb = {
 inherit (cfg.influxdb) scheme host port token org bucket;
 tls.insecure_skip_verify = cfg.influxdb.tlsSkipVerify;
 };
 };
 log = {
 level = cfg.logLevel;
 };
 };
 };
}

I also noticed that I had hard-coded the path to the binaries in the rendered systemd units. You might recall that each Nix package contains meta block, which adds information about the package such as the license, maintainer, etc. For each of the packages, I added a mainProgram attribute which enables the use of something like lib.getExe pkgs.scrutiny rather than ${pkgs.scrutiny}/bin/scrutiny.

1
2
3
4
5
6
7


{
 # ...
 meta = {
 mainProgram = "scrutiny";
 # ...
 };
}

Finally, I wanted to simplify the packaging of the Go programs slightly. In the previous post I used buildPhase to override the default build behaviour of the buildGoModule function. I later discovered the subPackages attribute, which enabled me to achieve the same effect without overriding the process manually. You can see the commit with all the changes on Github, but below is a (slightly modified) preview of the simplified collector derivation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


{ pkgs, lib, ... }:
pkgs.buildGoModule rec {
 # (source, name, version, vendor hash omitted)

 subPackages = "collector/cmd/collector-metrics/collector-metrics.go";

 buildInputs = with pkgs; [ makeWrapper ];

 CGO_ENABLED = 0;
 ldflags = [ "-extldflags=-static" ];
 tags = [ "netgo" "static" ];

 installPhase = ''
 mkdir -p $out/bin
 cp $GOPATH/bin/collector-metrics $out/bin/scrutiny-collector-metrics
 wrapProgram $out/bin/scrutiny-collector-metrics \
 --prefix PATH : ${lib.makeBinPath [ pkgs.smartmontools ]}
 '';

 # (meta block omitted)
}

Contributing To nixpkgs #

With these changes made, it was time to open a pull request. The NixOS community provide some contributing guidelines which are worth a read before you start.

There was a structural transition in the pkgs directory lately, so this was my first contribution under that new structure. There was a detailed talk about that transition at Nixcon if you’d like more information.

I needed to do the following:

Create the scrutiny and scrutiny-collector packages
Add the scrutiny module
Add the tests
Update the release notes for the next release of NixOS

Which translated into the following stack of commits on PR #289934:

There were a few other minor changes, such as how input packages (like smartmontools and makeWrapper) are passed to the packages. In the nixpkgs repository, packages are passed directly to the derivations, rather than pkgs being passed as a top-level argument and packages being referred to by pkgs.smartmontools etc.

I also learned something new about module configuration. In the first iteration I had set the default value of services.scrutiny.collector.enable to the value of services.scrutiny.enable, meaning that the collector would be enabled automatically when Scrutiny was enabled if no other configuration was specified. It transpires that this is not permitted in nixpkgs, and resulted in this CI failure. I updated the module with a subtle API change, meaning that consumers of the module will now need to do the following to get the same behaviour.

1
2
3
4
5
6


{
 services = {
 scrutiny.enable = true;
 scrutiny.collector.enable = true;
 };
}

Review Feedback #

From the first review I learned about the lib.types.nullOr meta-type which enables the specification of optionally null configuration options in a NixOS module.

What followed was a suggestion to embrace a different configuration format according to RFC42: a proposal for structured module configuration which is more rigorously type checked, and more flexible as new options are added to the underlying workload.

Many NixOS modules solve this problem today with options like extraConfig or configLines, which append strings to the end of any options-generated configuration file. What RFC42 proposes is a hybrid approach whereby certain options are well-defined (and thus feature in the documentation), but additional options can be injected as needed. The has the nice side-effect that modules do not need to support dozens of options which can become a burden to maintain and test over time, but consumers can still make use of new features and options that are introduced to the underlying workload.

You can see the full resulting options definition on Github, and small annotated excerpt below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


{
 services.scrutiny = {
 enable = lib.mkEnableOption "Enables the scrutiny web application.";

 # ...

 settings = lib.mkOption {
 description = lib.mdDoc ''
 Scrutiny settings to be rendered into the configuration file.

 See https://github.com/AnalogJ/scrutiny/blob/master/example.scrutiny.yaml.
 '';

 default = { };

 # This is the key to RFC42 - defining `settings` as type `submodule`.
 type = lib.types.submodule {
 # This ensures that attrsets, lists, etc. are rendered as JSON
 freeformType = (pkgs.formats.yaml { }).type;

 # The following are examples of "well-defined" options with
 # formal types and metadata.
 options.web.listen.port = lib.mkOption {
 type = lib.types.port;
 default = 8080;
 description = lib.mdDoc "Port for web application to listen on.";
 };

 options.web.listen.host = lib.mkOption {
 type = lib.types.str;
 default = "0.0.0.0";
 description = lib.mdDoc "Interface address for web application to bind to.";
 };

 # ...
 };
 };
}

This enables the following user configurations:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


{
 services.scrutiny = {
 enable = true;
 settings = {
 # These options are well-defined in the module with types,
 # examples, descriptions and are rendered in the docs.
 web.listen = {
 port = 8080;
 host = "100.64.12.34";
 };

 # These are not, but will be rendered still as JSON and enable
 # people to make use of options supported by Scrutiny, but
 # not directly by the module.
 notify = {
 urls = [
 "discord://token@channel"
 ];
 };
 };
 };
}

I was impressed by the simplicity of the approach, and subsequently adopted the same for the collector, which now has the ability to take configuration through the services.scrutiny.collector.settings attribute.

One aspect of Scrutiny’s design that helped is the ability to take configuration either through from file, or through environment variables, or both! I chose to supply some fixed options as environment variables to the systemd services, and allow the rest of the configuration to be set using the generated file.

@JulienMalka reviewed next, suggesting changes to ensure that various preInstall and postInstall hooks were being called where I was overriding installPhase, though what followed from @katexochen’s review was a simplification that reduced this need slightly.

@katexochen suggested a couple of nice changes, among which was the ability to drop explicitly mentioning the netgo tag in the Go packages, since CGO_ENABLED is disabled already, this implies using the Go internal version of many libraries. They also suggested moving the install of the frontend pieces to postInstall, rather than overriding the standard installPhase.

And finally @SuperSandro2000 reviewed with some naming changes to simplify the package definitions and reduce repetition. I also noticed that I was unnecessarily declaring the frontend definition using the rec keyword. Once these were fixed the PR was merged!

Contribution Tips #

I’m still relatively new to this - you can see that my list of contributions is not extensive, but I’ve been through the process of adding, updating and refining packages and modules a few times.

Take this advice for what it is, but I hope you find it helpful:

Read the contribution guidelines: This is probably the best start. Try to follow the guidance as best you can, and take a look at some other recently closed Pull Requests to see if you can pick up on any patterns.
Be humble: Everyone is new when they start. You might find some reviews a little terse. Rest assured that this is only because the maintainers are incredibly busy. Assume good intentions, remain polite and respectful. Nixpkgs is one of the busiest Github repositories I’ve seen, and it’s largely maintained by volunteers who often have limited time to give for a huge number of contributions.
Be patient: Don’t expect a review immediately. If you’re looking for a preliminary review, feel free to ping me on Mastodon and I’ll try to offer any guidance I can.
Use the Discourse: Because of the volume of PRs, the community run ongoing “PRs Ready for Review” threads on Discourse which you can use to attract reviewers if your PR has been stale a while. It’s also a great place to ask for help.

Once merged, commits trickle through the various branches of the repository. You can track the progress of your PR in that process once it’s merged using the PR Tracker. You can see an example of this by tracking the Scrutiny PR:

Summary & Thanks #

My Scrutiny packages, module and tests landed in nixpkgs approximately 4 days after I submitted the pull request. As ever, I received a bunch of helpful reviews from the community, and Scrutiny is now easier than ever to consume on NixOS.

You can see the finished components here:

If you were previously using my flake to consume Scrutiny, you’ll now see a deprecation warning. To migrate over, import the module from unstable in your flake and ensure that you explicitly enable the collector:

1
2
3
4
5
6


{
 services = {
 scrutiny.enable = true;
 scrutiny.collector.enable = true;
 };
}

Before I wrap up, I’d like to thank all those people who took the time to review the PR, and especially @JulienMalka and @RaitoBezarius who have helped me through countless contributions over the past months and been incredibly responsive to my questions.

Give it a go and let me know how you get on!

Packaging Scrutiny for NixOS

Fri, 16 Feb 2024 00:00:00 +0000

Update: Since writing this post, I’ve contributed Scrutiny upstream to nixpkgs/NixOS. You can see the write up of that effort here.

Introduction #

In a recent (well, recent-ish) episode of the Self Hosted Show, there was some talk of a hard drive monitoring tool called Scrutiny. Scrutiny is a hard drive monitoring tool that exposes S.M.A.R.T data in a nice, clean dashboard. It gathers that S.M.A.R.T data using the venerable smartd, which is a Linux daemon that monitors S.M.A.R.T data from a huge number of ATA, IDE, SCSI-3 drives. The code is available on Github.

The aim of running such monitoring is to detect and replace failing hard drives before they cause an outage, or any data loss. Depending on the firmware of a drive, there are potentially hundreds of S.M.A.R.T attributes that can be collected, so it can be hard to understand which to pay attention to.

As good as smartd is, it has some shortcomings such as not recording historical data for each attribute, and relying upon value thresholds that are set by the manufacturers of the drive. Scrutiny aims to provide historical tracking of attribute readings, and thresholds based on real-world data from Backblaze.

This all looked very compelling, but it hasn’t been packaged for NixOS as far as I can tell. This post is quite “code heavy” and is intended as a bit of a deep-dive/walkthrough for people who are new to packaging in Nix. I’m certainly no expert here, and if you spot something I’ve done wrong - I’d love to hear about it!

Below is a screenshot of Scrutiny’s dashboard:

Architecture #

In order to get Scrutiny up and running, there were a few components that all needed packaging separately.

Scrutiny itself is made up of the following:

A web application backend - written in Go
A web application frontend - written in NodeJs/Angular
A collector service - written in Go

The web application is the dashboard that you’ll see the pretty screenshots of. The collector service is designed to be run on an interval to collect information from smartd, and send it to the web application’s API.

Scrutiny relies upon InfluxDB to store data about smart attributes in a timeseries.

Thinking about how to structure things, I decided that I would build two separate Nix packages: one for the dashboard and UI, and another for the collector. It seems one could run the collector and dashboard on different machines, so this seemed like a logical split.

Packaging for NixOS #

Packaging the Dashboard #

I decided to start with the web application. Because of how the project is laid out, this means combining two separate derivations based built from the same source code: one for the UI and one for the backend. I began by creating a file that would carry common attributes for each of the derivations, such as the version, repository information, hash, etc.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


# common.nix
{ pkgs , ... }:
let
 name = "scrutiny";
 version = "0.7.2";
in
{
 inherit name version;

 # Fetch the source code from Github, referring to
 # the version required by Git tag.
 repo = pkgs.fetchFromGitHub {
 owner = "AnalogJ";
 repo = name;
 rev = "v${version}";
 sha256 = "sha256-UYKi+WTsasUaE6irzMAHr66k7wXyec8FXc8AWjEk0qs=";
 };

 # ...
}

I then started packaging the frontend:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36


{ pkgs, ... }: {
 # Import some common attributes such
 # as name, version, src repo
 common = import ./common.nix { inherit pkgs; };
 inherit (common) name version repo vendorHash;

 # Create a package for the Javascript frontend
 frontend = pkgs.buildNpmPackage rec {
 inherit version;
 pname = "${name}-webapp-frontend";
 src = "${repo}/webapp/frontend";

 # This hash is generated at build time, and uniquely identifies
 # the cache of NodeJS dependencies used for the build.
 npmDepsHash = "sha256-M8P41LPg7oJ/C9abDuNM5Mn+OO0zK56CKi2BwLxv8oQ=";

 # Override the build phase to match the
 # upstream build process.
 buildPhase = ''
 runHook preBuild
 mkdir dist
 npm run build:prod --offline -- --output-path=dist
 runHook postBuild
 '';

 # Copy the output of the compiled javascript bundle
 # and site assets to the package output.
 installPhase = ''
 runHook preInstall
 mkdir $out
 cp -r dist/* $out
 runHook postInstall
 '';
 };
 # ...
}

This is a relatively simple derivation, mostly thanks to the magic of the buildNpmPackage function. This helper function takes care of creating an offline cache containing all of the NodeJS dependencies required to build the frontend. Nix builds are always done in an offline environment to help ensure reproducibility, so source code and dependencies are fetched (and hashed) early in the process before any software is actually built.

I chose to override the build phase to match the process used by the upstream Makefile. The result of this derivation is a package named scrutiny-webapp-frontend, which contains just the built output from the npm run build:prod command.

Next up was the dashboard backend. Another pleasingly simple derivation thanks to some help from buildGoModule:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


pkgs.buildGoModule rec {
 # The vendor hash is used for multiple packages, and
 # thus centralised and imported here.
 inherit version vendorHash;
 pname = "${name}-webapp-backend";

 # Use the source block defined in 'common.nix'
 src = repo;

 CGO_ENABLED = 0;

 # Override the build phase to ensure the correct
 # binary is built. This project ships both Go applications
 # and the NodeJS frontend in the same repository, so some
 # specificity is required.
 buildPhase = ''
 runHook preBuild
 go build \
 -o scrutiny-web \
 -ldflags="-extldflags=-static" \
 -tags "static netgo" \
 ./webapp/backend/cmd/scrutiny
 runHook postBuild
 '';

 # Add the built binary to the package output, as well
 # as the build output from the frontend.
 installPhase = ''
 mkdir -p $out/bin $out/share/scrutiny
 cp scrutiny-web $out/bin/scrutiny-web
 cp -r ${frontend}/* $out/share/scrutiny
 '';
}

This one is a little more interesting. There are some commonalities, such as setting the vendorHash to ensure the correct Go dependencies are used (imported from common.nix in this case), and overriding the build to match the upstream process and ensure the right binary is built.

Where things differ is in the install phase, where the contents of the scrutiny-webapp-frontend derivation is copied into the output of this derivation - which will ultimately result in a single Nix package (named scrutiny-webapp-backend) which will contain both the frontend and backend components of the application. This is ultimately exposed in an overlay as a package simply named scrutiny.

You can see the finished product (with additional package metadata) in app.nix and common.nix on Github.

Packaging the Collector #

The collector is just a single, statically compiled Go binary, and as such the derivation very much resembles that of the web application backend above:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39


{ pkgs, lib, ... }:
let
 # Again import common attributes from 'common.nix'
 common = import ./common.nix { inherit pkgs; };
 inherit (common) name version repo vendorHash;
in
pkgs.buildGoModule rec {
 inherit version vendorHash;
 pname = "${name}-collector";
 src = repo;

 # We'll create a wrapper script for the
 # collector, which makeWrapper helps with.
 buildInputs = with pkgs; [ makeWrapper ];

 CGO_ENABLED = 0;

 # Override the build phase to ensure the correct
 # binary is built.
 buildPhase = ''
 runHook preBuild
 go build \
 -o scrutiny-collector-metrics \
 -ldflags="-extldflags=-static" \
 -tags "static netgo" \
 ./collector/cmd/collector-metrics
 runHook postBuild
 '';

 # Install both the binary, and a generated wrapper script
 # which ensures that 'smartctl' is in the PATH of the collector.
 installPhase = ''
 mkdir -p $out/bin
 cp scrutiny-collector-metrics $out/bin/scrutiny-collector-metrics

 wrapProgram $out/bin/scrutiny-collector-metrics \
 --prefix PATH : ${lib.makeBinPath [ pkgs.smartmontools ]}
 '';
}

Of interest here is the installPhase. The collector works by invoking smartctl to scrape information from smartd. Scrutiny itself expects that tool to be readily available in it’s PATH, and to accomplish that I used the makeWrapper package to create a wrapper script that ensures scrutiny-collector-metrics is executed with the PATH correctly set such that smartctl can be found.

The final derivation for the collector can be seen in collector.nix on Github.

Writing a NixOS Module #

While I now had functioning packages for all of Scrutiny’s components, for them to function correctly on a machine there are a few things that need to be in place:

The dashboard package must be installed and started
The collector collector package must be installed and started
InfluxDB must be installed and started
Scrutiny dashboard must be configured to speak to the host’s InfluxDB
smartd must be installed and running

This sort of challenge is exactly what the NixOS modules system aims to solve. Modules take a set of configuration attributes, and convert them into rendered system configuration in the form of systemd units, configuration files and more.

It’s tempting to try to support all possible configurations when writing a module, but I generally prefer to start small, and support only the configuration I need for my use-case. This keeps things easy to test (and leaves some fun for the future!). In this case, the module would be first installed on a server which hosts a set of services behind the Traefik reverse proxy. To work out what configuration I wanted to provide, I looked at the upstream’s example config file. The important things that stood out to me for consideration were:

The location of the web UI files to serve
The host/port of the InfluxDB instance
The “base path” - Scrutiny will be exposed at https://<tailscale node>/scrutiny

As mentioned before - each NixOS module consists of some options, and some rendered config. The options block for my Scrutiny web app looks like the below snippet. I’ve omitted comments this time, as I think the language is quite descriptive here:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48


{
 let
 cfg = config.services.scrutiny;
 in
 options = {
 services.scrutiny = {
 enable = lib.mkEnableOption "Enables the scrutiny web application";

 # Use the `scrutiny` package by default.
 package = lib.mkPackageOptionMD pkgs "scrutiny" { };

 port = lib.mkOption {
 type = lib.types.port;
 default = 8080;
 description = lib.mdDoc "Port for web application to listen on";
 };

 host = lib.mkOption {
 type = lib.types.str;
 default = "0.0.0.0";
 description = lib.mdDoc "Interface address for web application to bind to";
 };

 basepath = lib.mkOption {
 type = lib.types.str;
 default = "";
 description = lib.mdDoc ''
 If Scrutiny will be behind a path prefixed reverse proxy, you can override this
 value to serve Scrutiny on a subpath.

 Do not include the leading '/'.
 '';
 };

 openFirewall = lib.mkOption {
 type = lib.types.bool;
 default = false;
 description = "Open the default ports in the firewall for Scrutiny.";
 };

 logLevel = lib.mkOption {
 type = lib.types.enum [ "INFO" "DEBUG" ];
 default = "INFO";
 description = lib.mdDoc "Log level for Scrutiny.";
 };
 };
 };
}

This provides some basic configuration for the attributes I care about - noticeably missing is any advanced configuration for InfluxDB (such as org, bucket, token), and and any ability to configure notifications which Scrutiny supports through the excellent shoutrrr library. These things will come later.

I needed a convenient way to convert this Nix-native configuration format into the right format for Scrutiny - I wrote a small Nix function to help with that, which takes configuration elements from the options defined above, and writes them into a small YAML file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


{
 mkScrutinyCfg = cfg: pkgs.writeTextFile {
 name = "scrutiny.yaml";
 text = ''
 version: 1
 web:
 listen:
 port: ${builtins.toString cfg.port}
 host: "${cfg.host}"
 basepath: "${if cfg.basepath != "" then "/" + cfg.basepath else ""}"

 database:
 location: "/var/lib/scrutiny/scrutiny.db"

 src:
 frontend:
 path: "${pkgs.scrutiny}/share/scrutiny"

 log:
 level: "${cfg.logLevel}"
 '';
 };
}

Note that this hard-codes some key elements, such as the path of the web application assets which are shipped as part of the scrutiny package.

Let’s take a look at the part of the module which turns these options into a rendered system configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


{
 config = {
 # If scrutiny is enabled, also enable InfluxDB with default settings
 services.influxdb2 = lib.mkIf cfg.enable {
 enable = true;
 };

 # Open the relevant ports in the system firewall if configured
 networking.firewall = lib.mkIf cfg.openFirewall {
 allowedTCPPorts = [ cfg.port ];
 };

 # If Scrutiny is enabled, create a systemd unit to start it
 systemd = {
 services = {
 scrutiny = lib.mkIf cfg.enable {
 description = "Hard Drive S.M.A.R.T Monitoring, Historical Trends & Real World Failure Thresholds";
 wantedBy = [ "multi-user.target" ];
 after = [ "network.target" ];
 serviceConfig = {
 # Don't run the application a root - create a dynamic user as it starts
 DynamicUser = true;
 # Start the application with a config rendered by the helper function
 ExecStart = "${cfg.package}/bin/scrutiny-web start --config ${mkScrutinyCfg cfg}";
 Restart = "always";
 StateDirectory = "scrutiny";
 StateDirectoryMode = "0750";
 };
 };
 };
 };
 };
}

That’s enough to get the dashboard started, but it doesn’t take care of starting the collector. For that, I added a couple more configuration options to the module:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


{
 let
 collectorCfg = config.services.scrutiny.collector;
 in
 {
 options = {
 services.scrutiny = {
 # ...
 collector = {
 enable = lib.mkOption {
 type = lib.types.bool;
 default = cfg.enable;
 description = lib.mdDoc "Enables the scrutiny collector";
 };

 # Use the `scrutiny-collector` package by default.
 package = lib.mkPackageOptionMD pkgs "scrutiny-collector" { };

 endpoint = lib.mkOption {
 type = lib.types.str;
 default = "http://${cfg.host}:${builtins.toString cfg.port}/${cfg.basepath}";
 description = lib.mdDoc "Scrutiny app API endpoint for sending metrics to.";
 };

 interval = lib.mkOption {
 type = lib.types.str;
 default = "15m";
 description = lib.mdDoc ''
 Interval on which to collect information about disks.

 Examples: 15m, 10s, 2h.
 '';
 };
 };
 };
 };
 }
}

With that configuration in place, I needed to adjust the rendered configuration to include starting the collector. The collector is designed to run on an interval to post metrics to the dashboard. The upstream achieves this with cron, but I decided to use systemd timers:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39


{
 config = {
 # ...
 # If the collector is enabled, enable smartd so that the
 # collector can scrape metrics from it
 services.smartd = {
 enable = collectorCfg.enable;
 extraOptions = [ "-A /var/log/smartd/" "--interval=600" ];
 };

 systemd = {
 services = {
 # Setup a systemd unit to start the collector
 scrutiny-collector = lib.mkIf collectorCfg.enable {
 description = "Scrutiny Collector Service";
 environment = {
 COLLECTOR_API_ENDPOINT = "${collectorCfg.endpoint}";
 };
 serviceConfig = {
 Type = "oneshot";
 ExecStart = "${cfg.collector.package}/bin/scrutiny-collector-metrics run";
 };
 };
 };

 # Set up a systemd timer to trigger the collector at an
 # interval (default 15m).
 timers = {
 scrutiny-collector = lib.mkIf collectorCfg.enable {
 timerConfig = {
 OnBootSec = "5m";
 OnUnitActiveSec = "${collectorCfg.interval}";
 Unit = "scrutiny-collector.service";
 };
 };
 };
 };
 };
}

And that’s it! You can see the final module all stitched together on Github.

Automated Testing #

To me, a super exciting part of the NixOS ecosystem is its automated testing framework, which is used to great effect for testing hundreds of packages end-to-end before they’re released to the various NixOS channels. The NixOS testing framework provides a set of helpers for spawning fresh NixOS virtual machines, and ensuring that applications can start, services have the right side effects, etc.

I wanted to write a simple test to validate that Scrutiny will continue to work as I update my flake. In my view, the test needed to:

Ensure that the packages could be built
Ensure that when services.scrutiny.enable = true is set, the services start
Ensure that the dashboard app renders the UI
Ensure that the metrics collector can speak to the dashboard

Most of these are relatively trivial - one piece that’s a little harder is testing that the UI renders correctly. The application is rendered using Javascript client-side, so a simple curl won’t get us the results we’re expecting. I had previously used selenium for this purpose when I submitted a test for the LXD UI, so I chose to use that approach again.

The test code can be seen below, or on Github:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69


{
 name = "scrutiny";
 nodes = {
 # Configure a NixOS virtual machine for the test
 machine = { self, pkgs, lib, ... }: {
 # Ensure that my NixOS module is imported
 imports = [ self.nixosModules.scrutiny ];
 # Add the package overlay from my flake so that `pkgs.scrutiny` resolves.
 nixpkgs.overlays = [ self.outputs.overlays.additions ];

 # Configure the VM to use my module and enable it
 services.scrutiny.enable = true;


 environment.systemPackages =
 let
 # A selenium script that fetches the dashboard using geckodriver
 seleniumScript = pkgs.writers.writePython3Bin "selenium-script"
 {
 libraries = with pkgs.python3Packages; [ selenium ];
 } ''
 from selenium import webdriver
 from selenium.webdriver.common.by import By
 from selenium.webdriver.firefox.options import Options
 from selenium.webdriver.support.ui import WebDriverWait

 options = Options()
 options.add_argument("--headless")
 service = webdriver.FirefoxService(executable_path="${lib.getExe pkgs.geckodriver}") # noqa: E501

 driver = webdriver.Firefox(options=options, service=service)
 driver.implicitly_wait(10)
 driver.get("http://localhost:8080/web/dashboard")

 wait = WebDriverWait(driver, 60)

 # Look for some elements that should be rendered by the Javascript bundle in the UI
 assert len(driver.find_elements(By.CLASS_NAME, "mat-button-wrapper")) > 0
 assert len(driver.find_elements(By.CLASS_NAME, "top-bar")) > 0

 driver.close()
 '';
 in
 with pkgs; [ curl firefox-unwrapped geckodriver seleniumScript ];
 };
 };
 # This is the test code that will check if our service is running correctly:
 testScript = ''
 start_all()

 # Wait for InfluxDB to be available
 machine.wait_for_unit("influxdb2")
 machine.wait_for_open_port(8086)

 # Wait for Scrutiny to be available
 machine.wait_for_unit("scrutiny")
 machine.wait_for_open_port(8080)

 # Ensure the API responds as we expect
 output = machine.succeed("curl localhost:8080/api/health")
 assert output == '{"success":true}'

 # Start the collector service to send some metrics
 collect = machine.succeed("systemctl start scrutiny-collector.service")

 # Ensure the application is actually rendered by the Javascript
 machine.succeed("PYTHONUNBUFFERED=1 selenium-script")
 '';
}

This relatively short code snippet will take care of:

Building a dedicated virtual machine according to the machine spec
Starting the VM
Running the tests inside the VM (specified in `testScript)
Collecting the results

I ended up adding this check in a Github Actions workflow so that it’s run each time I make a change to my flake.

Try It! #

You can give this a go too! If you’ve got a NixOS machine, or perhaps a virtual machine you’ve been experimenting with, then you need only add the following to your flake:

1
2
3
4
5
6
7
8
9


{
 inputs = {
 nixpkgs-unstable.url = "github:nixos/nixpkgs/nixos-unstable";
 jnsgruk.url = "github:jnsgruk/nixos-config";
 jnsgruk.inputs.nixpkgs.follows = "nixpkgs-unstable";
 # ...
 }
 # ...
}

And in your NixOS machine configuration:

1
2
3
4
5
6


# Where 'inputs' is the inputs attribute of your flake
{ inputs, ... }: {
 imports = [ inputs.jnsgruk.nixosModules.scrutiny ];
 nixpkgs.overlays = [ inputs.jnsgruk.overlays.additions ];
 services.scrutiny.enable = true;
}

The next time you rebuild your system, you should be able to browse to http://localhost:8080 and see some hard drive metrics!

What’s Next? #

This was a pretty quick exercise - I’d estimate at about 3 hours in total. No doubt I will find some bugs, but already in my mind are the following future improvements:

Add some more tests that exercise more of the config options
Enable notifications configuration in the module
Submit the packages/module to the upstream or to nixpkgs

I also need to go and investigate this rather sad looking hard disk…

That’s all for now! Thanks for reading if you got this far!

NixOS on Jon Seager

The Immutable Linux Paradox

What is an immutable Linux distribution? #

Why immutability? #

The immutability paradox #

Approaches to immutability #

Fedora Silverblue / CoreOS / EndlessOS (ostree) #

RHEL “Image Mode” (bootc) #

NixOS #

Ubuntu Core #

Summary #

From NixOS to Ubuntu

Recap: Why NixOS? #

Adopting “JFUI” #

Desktop Environment #

GNOME Extensions #

Editors #

Terminal #

Software Availability #

Configuration Management #

So What About Nix? #

Summary #

Packaging the Multipass Flutter GUI for NixOS

Introduction #

Housekeeping #

Restructuring #

Using symlinkJoin #

Building the GUI #

Update Script #

Summary #

Experimenting with Rust, Nix, K6 and Parca

Introduction #

Learning Rust #

Rewriting gosherve in Rust #

Basic Load Testing with k6 #

Nix-ified Load Testing #

Defining Test VMs #

Wrapping k6 #

Automating Test Execution #

Initial Load Test Results #

Profiling gosherve with Parca #

Instrumenting gosherve #

Initial optimisations #

Reducing Allocations #

Profiling servy? #

Final Results & Comparisons #

Summary #

Hot Tub Monitoring with Home Assistant and ESPHome

Introduction #

Energy Usage Monitoring #

Temperature Sensor: Initial Solution #

Temperature Sensor: Homemade Solution #

Connecting the ESP32 #

ESPHome Support #

ESPHome Dashboard #

Creating a Firmware #

3D Printing an Enclosure #

Home Assistant Dashboard #

Summary #

Writing a Home Assistant Core Integration: Part 2

Introduction #

Development Setup #

Integration Basics #

File Structure #

manifest.json #

__init__.py #

config_flow.py #

coordinator.py #

climate.py #

Testing #

Docs & Brand #

Contribution Process #

Results #

Summary #

Writing a Home Assistant Core Integration: Part 1

Introduction #

Designing the library #

Upstream API #

Basic Requirements #

Outline design/experience #

Fedora Silverblue / CoreOS / EndlessOS (`ostree`) #

RHEL “Image Mode” (`bootc`) #

Using `symlinkJoin` #

Rewriting `gosherve` in Rust #

Basic Load Testing with `k6` #

Wrapping `k6` #

Profiling `gosherve` with Parca #

Instrumenting `gosherve` #

Profiling `servy`? #

`manifest.json` #

`init.py` #

`config_flow.py` #

`coordinator.py` #

`climate.py` #

`uv` #

`pydantic` #

`ruff` #

Contributing to `nixpkgs` #

Enable `lanzaboote` #