Nix on Jon Seager

Ubuntu Summit 25.10: Personal Highlights

Sun, 02 Nov 2025 00:00:00 +0000

This article was originally posted on the Ubuntu Discourse, and is reposted here. I welcome comments and further discussion in that thread.

I recently had the privilege of attending the Ubuntu Summit 25.10. Ubuntu Summits have a relatively long history. Some years ago Canonical ran the ‘Ubuntu Developer Summits (UDS)’, but recently the events were brought back and reimagined as the ‘Ubuntu Summit’.

For the most recent Summit, we tried out a new format. We invited a select few folks to come and give talks at our London office, with a small in-person crowd. In addition, the event was livestreamed, and we encouraged people to host “watching parties” across the world as part of Ubuntu Local Communities (LoCos).

While Ubuntu may feature in the name, the event does not require talks to be centred on Ubuntu, and in fact is aiming to draw contributions from our partners and from right across the open source community, whether or not the content is relevant to Ubuntu or Canonical - it’s designed to be a showcase for the very best of open source, and this year I felt that the talks were of a particularly high calibre.

In this post I’ll highlight some of my favourite talks, in no particular order! If any of these catch your interest, you can see when they were aired and catch-up on the Day 1 and Day 2 streams.

DOOM in Space #

What a way to kick off the Summit! DOOM in Space was a talk given by Ólafur Waage, who introduced himself as a “professional keyboard typist”!

The talk was immediately after Mark Shuttleworth’s opening remarks, and covered his journey in getting DOOM to run on the European Space Agency’s OPS-SAT satellite. DOOM has famously been ported to many devices, though some were only questionably “running” the game.

Ólafur covered how he became involved in the project, and the unique approach they needed to take to guarantee success, since they would only get a very limited amount of time in order to conduct their “experiment” on the satellite.

Of particular note was the work done to integrate imagery from the OPS-SAT’s onboard camera into the game, which involved some clever reassigning of colors in the game’s original palette to more faithfully represent the imagery taken from the camera in-game.

Infrastructure-Wide Profiling of Nvidia CUDA #

This talk was given by Frederic Branczyk, CEO and Founder of Polar Signals. Canonical has partnered with Polar Signals a couple of times in recent years. They were part of our journey to enabling frame pointers by default on Ubuntu, and many of our teams have been using their zero-instrumentation eBPF profiler.

While CPU profiling has been commonplace for developers for many years, giving the ability to analyse CPU and memory-bound workloads, profiling GPU workloads has been less prominent, and is particularly difficult in production.

Polar Signals advocate for “continuous profiling”, which means running a profiler at all times, on all nodes, in production. The benefit of this is that when an issue occurs, you don’t have to set up a profiler and try to reproduce the issue - you already have the data. It also negates the uncertainty of the impact a profiler might have on the code during reproduction. This would have been difficult with traditional profiling tools, but with technologies like eBPF, the overhead of the profiler is incredibly low compared to the potential performance gains from acting on the data it produces.

In this talk, Frederic outlined the work they have done bringing infrastructure-wide profiling of CUDA workloads into Polar Signals Cloud. Their approach combines the CUPTI profiling API with USDT probes and eBPF into a pipeline, relying upon the ability to inject a small library into CUDA workloads using the CUDA_INJECTION64_PATH without modification.

You can see more details on their website.

Inference Snaps #

This talk served as the first public announcement of Inference Snaps from Canonical, which represents a few months of working combining many of the new technologies behind Snaps.

As Large Language Models continue to gain pace along with the rest of the AI community, silicon manufacturers are increasingly including dedicated hardware in commodity CPUs and GPUs, as well as shipping dedicated accelerators for some workloads.

AI models often need to be tuned in some way in order to work optimally - for example quantisation which aims to reduce the computational memory costs of running inference on a given model.

Inference snaps provide a hassle-free mechanism for users to obtain the “famous model” they want to work with, but automatically receive a version of that model which is optimised for the silicon in their machine, removing the need to spend hours on HuggingFace trying to identify the correct model to download that matches with their hardware.

Using our extensive partner network, we’ll continue to work with multiple silicon vendors to ensure that models are available for the latest hardware as it drops, and provide a consistent experience to Ubuntu users that wish to work with AI.

Find out more in the announcement.

Nøughty Linux: Ubuntu’s Stability Meets Nixpkgs’ Freshness #

This talk was a bit of a guilty pleasure for me! Delivered by Martin Wimpress (wimpy), the audience were shown how they could take a stock Ubuntu Server deployment, and use a collection of scripts to layer a cutting-edge GUI stack on top using Nixpkgs.

Wimpy outlined his motivation as wanting to rely upon the stable kernel and hardware support offered by Ubuntu, but wanting to be more experimental with his desktop environment and utilities - preferring a tiling window management experience.

Having spent some years on NixOS, Wimpy was recently required to run a security “agent” for work, which was very difficult to enable on NixOS, but worked out of the box on Ubuntu. Recognising the need to make the switch, he was reluctant to move away from the workflow he’d built so much muscle-memory around - and so Nøughty Linux was born!

Nøughty Linux is not a Linux distribution, rather a set of configurations for an Ubuntu Server machine. It utilises nix-system-graphics and system-manager and is actually very similar to a configuration I ran in my own nixos-config repository for my laptop for a while - though Wimpy has chased down significantly more of the papercuts than I did!

Are we stuck with the same Desktop UX forever? #

Scott Jenson delivered an incredibly engaging talk in which he posited that desktop user experience has somewhat stagnated, and worse that many of the patterns we’ve become used to on the desktop are antiquated and unergonomic.

The crux of the talk was to focus on user experience, rather than user interfaces - challenging developers to think about how people learn, and how desktops could benefit more from design affordances by rethinking some critical elements such as window management or text editing.

Using his years of experience at Apple, Symbian and Google, Scott delivered one of the most engaging conference talks I’ve seen, and I thoroughly recommend watching it on our YouTube channel!

Honorable Mentions #

In addition to the talks above, it was a delight to meet Mark Schoolderman from the Trifecta Tech Foundation in-person, who led the work on sudo-rs as part of our “Oxidising Ubuntu” story, and interesting to hear about the value the project derived from Ubuntu’s Main Inclusion Review process as part of landing sudo-rs in main.

Equally, I was delighted that Samuele Kaplun from Proton could join us to talk about the work we’ve been doing together on bringing first-class Snap packages for Proton Mail, Proton VPN, Proton Pass and Proton Authenticator to the Snap store, and their reasons for choosing Snaps, adventures with confinement, and more.

I was delighted to see Craig Loewen and Clint Rutkas present on their journey open sourcing the Windows Subsystem For Linux (WSL), which represents a growing proportion of Ubuntu users, and a key bridge to open source development for many.

Finally, thank you to Utkarsh for this wonderful slide as part of his talk on Ubuntu Snapshot Releases:

Conclusion #

Overall, I found the Ubuntu Summit 25.10 a really enjoyable event, with talks that were uniformly high in quality, charisma and creativity. I’m pleased that Canonical has broadened the Summit’s reach and I hope it continues to serve as a platform to showcase the very best open source innovation.

Until next time!

Packaging the Multipass Flutter GUI for NixOS

Thu, 16 Jan 2025 00:00:00 +0000

Introduction #

The very first application I ever packaged in nixpkgs was Multipass. Multipass is, according to the website:

a CLI to launch and manage VMs on Windows, Mac and Linux that simulates a cloud environment with support for cloud-init.

I wrote in some detail about how I use Multipass and LXD on my workstation to create and manage VMs for testing and development. Last year Multipass shipped a new GUI written from the ground up in Flutter. It provides a clean, modern way to launch, manage and interact with VMs. When the GUI first shipped, I briefly attempted to get it to build with Nix, but some combination of my lack of knowledge, and the maturity of the Flutter tooling in nixpkgs at the time meant I never finished it.

As version 1.15.0 was in its release candidate stage, I decided to have another go!

This post will break down the process into steps, but if you’d like the tl;dr, take a look at the pull request.

Housekeeping #

Before getting started on packaging the GUI, I did some housekeeping on the Multipass package. First order of business was to simply bump the version to 1.15.0 and ensure the package still built, which it did.

Last year, RFC 140 was introduced to simplify the directory structure of nixpkgs, introducing a new pkgs/by-name directory which will (eventually) render pkgs/top-level/all-packages.nix useless. This new structure will make finding package definitions much easier, as they’ll all be arranged alphabetically by the first two characters of their name. Using the Multipass example, rather than pkgs/tools/virtualization/multipass/default.nix, the new preferred path would be pkgs/by-name/mu/multipass/package.nix. There was a talk at NixCon 2023 which summarised this nicely if you’d like more information.

As I was going to be carrying out some hefty work on Multipass by introducing the GUI, I took the opportunity to move to the new scheme as part of the changes.

Restructuring #

The Multipass package carries a few patches, and is already reasonably involved, so I decided that the best route forward was to create separate files for multipassd/multipass (the server & command line client binaries, written in C++, and already packaged) and multipass.gui (the new Flutter application). Thus the new structure for the package looks like so:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


.
# Derivation files
├── gui.nix
├── multipassd.nix
├── package.nix
# Lock file for Fluter app
├── pubspec.lock.json
# Update script for daemon/cli & GUI
├── update.sh
# Patches
├── cmake_no_fetch.patch
├── cmake_warning.patch
├── lxd_socket_path.patch
├── test_unreachable_call.patch
└── vcpkg_no_install.patch

Using `symlinkJoin` #

This package now takes a more unusual shape. Because the GUI is essentially a entirely separate application, written with a different toolkit in a different language, but I wanted the two to be installed as part of the overall multipass package, I decided to use two completely separate derivations and use symlinkJoin to bundle the two together. According to the manual, symlinkJoin:

can be used to put many derivations into the same directory structure. It works by creating a new derivation and adding symlinks to each of the paths listed. It expects two arguments, name, and paths. name (or alternatively pname and version) is the name used in the Nix store path for the created derivation. paths is a list of paths that will be symlinked. These paths can be to Nix store derivations or any other subdirectory contained within.

This is particularly useful for the Multipass package, because the Multipass GUI dynamically-links against libraries that are built as part of the multipassd derivation, and linking the two derivations in this way ensures that all those files are located correctly at runtime without any extra plumbing. This allowed me to keep the top-level package.nix relatively simple (annotated below):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55


{
# ... snip ...
}:
let
 name = "multipass";
 version = "1.15.0";

 # Fetch the source just once, to be used in both the multipassd and GUI derivations
 multipass_src = fetchFromGitHub {
 owner = "canonical";
 repo = "multipass";
 rev = "refs/tags/v${version}";
 hash = "sha256-RoOCh1winDs7BZwyduZziHj6EMe3sGMYonkA757UrIU=";
 fetchSubmodules = true;
 };

 # Shared metadata between derivations.
 commonMeta = {
 homepage = "https://multipass.run";
 license = lib.licenses.gpl3Plus;
 maintainers = with lib.maintainers; [ jnsgruk ];
 platforms = [ "x86_64-linux" ];
 };

 # Build the multipass server & client binaries from the shared source
 # and common metadata.
 multipassd = callPackage ./multipassd.nix {
 inherit commonMeta multipass_src version;
 };

 # Build the multipass GUI using shared source and common metadata.
 multipass-gui = callPackage ./gui.nix {
 inherit commonMeta multipass_src multipassd version;
 };
in
# Create the top-level `multipass` package using `symlinkJoin`
symlinkJoin {
 inherit version;
 pname = name;

 # Join both of the newly created derivations
 paths = [ multipassd multipass-gui ];

 # Ensure tests are run, and define a (new) update script.
 passthru = {
 tests = lib.optionalAttrs stdenv.hostPlatform.isLinux {
 inherit (nixosTests) multipass;
 };
 updateScript = ./update.sh;
 };

 meta = commonMeta // {
 description = "Ubuntu VMs on demand for any workstation";
 };
}

Building the GUI #

Normally the server, client and GUI binaries are all built as part of a single invocation of cmake. We can’t do that here, so the early code I put into the multipass package when I failed to package the GUI before still stood - essentially stopping cmake from trying to build the Flutter GUI:

1
2
3
4


{
 # We'll build the flutter application separately using buildFlutterApplication
 cmakeFlags = [ "-DMULTIPASS_ENABLE_FLUTTER_GUI=false" ];
}

Next, I created gui.nix which uses the pkgs.buildFlutterApplication helper. The derivation is annotated below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67


{
 # Passed in from the package.nix shown above
 commonMeta,
 multipass_src,
 multipassd,
 version,
 # ... snip ...
}:
flutter327.buildFlutterApplication {
 inherit version;
 pname = "multipass-gui";
 src = multipass_src;

 # Ensure we build in the correct directory for the GUI code.
 sourceRoot = "${multipass_src.name}/src/client/gui";

 # To make the builds as deterministic as possible, nixpkgs reads the pubspec.lock
 # file for the Flutter application and downloads/caches the correct versions of
 # flutter packages to be used in the build. It does this with `pub2nix`.
 #
 # Here I use a JSON representation of the pubspec.lock file committed into nixpkgs.
 pubspecLock = lib.importJSON ./pubspec.lock.json;

 # Some of the Flutter dependencies used are pulled from Github (branches) directly, so this
 # ensures that the versions/commits are pinned for future builds.
 gitHashes = {
 dartssh2 = "sha256-2pypKwurziwGLZYuGaxlS2lzN3UvJp3bRTvvYYxEqRI=";
 hotkey_manager_linux = "sha256-aO0h94YZvgV/ggVupNw8GjyZsnXrq3qTHRDtuhNv3oI=";
 system_info2 = "sha256-fly7E2vG+bQ/+QGzXk+DYba73RZccltdW2LpZGDKX60=";
 tray_menu = "sha256-riiAiBEms+9ARog8i+MR1fto1Yqx+gwbBWyNbNq6VTM=";
 window_size = "sha256-71PqQzf+qY23hTJvcm0Oye8tng3Asr42E2vfF1nBmVA=";
 xterm = "sha256-h8vIonTPUVnNqZPk/A4ZV7EYCMyM0rrErL9ZOMe4ZBE=";
 };

 # ... snip ...

 # The Multipass GUI relies on protobuf for API communication; ensure the
 # Dart protobuf definitions are compiled before we build.
 preBuild = ''
 mkdir -p lib/generated

 # Generate the Dart gRPC code for the Multipass GUI.
 protoc \
 --plugin=protoc-gen-dart=${lib.getExe protoc-gen-dart} \
 --dart_out=grpc:lib/generated \
 -I ../../rpc \
 ../../rpc/multipass.proto \
 google/protobuf/timestamp.proto
 '';

 # Libraries built by multipassd are linked dynamically at runtime; multipassd must
 # be installed for the GUI to function.
 runtimeDependencies = [ multipassd ];

 # Housekeeping to ensure the icons/desktop files are correctly installed.
 postFixup = ''
 mv $out/bin/multipass_gui $out/bin/multipass.gui

 install -Dm444 $out/app/multipass-gui/data/flutter_assets/assets/icon.png \
 $out/share/icons/hicolor/256x256/apps/multipass.gui.png

 cp $out/share/applications/multipass.gui.autostart.desktop \
 $out/share/applications/multipass.gui.desktop
 '';

 # ... snip ..
}

Update Script #

That was enough to ensure that the GUI worked correctly. My concern now lied with the maintenance of the package. The process for updating the derivation now involved:

Bumping the version string
Updating the source hashes
Updating the pubspec.lock.json file
Updating the gitHashes of the Flutter dependencies

This seemed a little convoluted and error prone to handle manually each time, so I decided to implement an update script for the package to take care of future version bumps. Thankfully nixpkgs has good provision for this, allowing package maintainers to specify passthru.updateScript to a derivation, which I did here.

I won’t post the full update script here, as it’s lots of unsightly bash! Essentially the script takes care of determining the latest version released on Github by using curl, then updates the pubspec.lock.json using a combination of curl/yj, then finally works through updating the source hash for the Multipass repo, gRPC fork it uses, and each of the git hashes for the Flutter dependencies.

This can be invoked from within the nixpkgs repo with:

1

nix-shell maintainers/scripts/update.nix --argstr package multipass

Summary #

Overall, this turned out cleaner than I expected, though it took me longer to figure out than this post might suggest. My hope is that this will a useful reference to anybody trying to package Flutter applications with Nix, and if you’re a user of the multipass package, check out the shiny new GUI that’s available to you! The GUI is available from 1.15.0 onwards, which at the time of writing means you’ll need to install the package from nixos-unstable or similar (sadly I didn’t land the change in time for NixOS 24.11).

In my opinion, the Multipass team have done a lovely job on the GUI - and for many users this will provide a much improved onboarding experience. Nice work 😎

Experimenting with Rust, Nix, K6 and Parca

Sun, 01 Dec 2024 00:00:00 +0000

Introduction #

Over the past couple of weeks, I’ve been teaching myself Rust. I don’t have a pressing need to write much Rust right now, but I’m intrigued by the promises of memory safety, and have been increasingly impressed at the quality of some of the software that the community produces. I also think that the concepts popularised by Rust, such as the borrow checker, will stick around in computing for many years to come and I’d like to have more hands-on experience with that.

The Rust language also encourages the ideas of safety and soundness - sound code is (approximately) code that can’t cause memory corruption or exhibit undefined behaviour. You can read more in this excellent post Safety and Soundness in Rust.

This blog post started out life as a post about Rust and my experience learning it, but I got interested in the performance of the server implementation I came up with and the post evolved into a post more about profiling and load testing!

Learning Rust #

It’s been a little while since I dug into a new programming language, so I thought I’d mention how I went about it. In general I’m someone who learns best “by doing”. I normally try to read through some of the basic concepts, then jump to a project that will enable me to exercise them quite quickly. Fortunately, Rust has an excellent official guide in the form of the Rust Book, which covers everything from the obligatory Hello, World!, to concurrency, memory safety, publishing packages on crates.io and more. I thoroughly recommend it.

I’ve also picked up a copy of Rust for Rustaceans which was recommended by a couple of different colleagues - I intend to work through this next.

So, after working through the Rust Book over the course of about a week in my spare time, I needed a project!

Rewriting `gosherve` in Rust #

In my first post on this blog I talked about a small Go project I wrote several years ago named gosherve. This was one of my first Go projects - a simple web server that can serve some static assets, and a set of short-links/redirects which are specified in a Github Gist. It’s been happily running my website for several years, and it felt like a small, but ambitious enough project for my first adventure into Rust - particuarly as over the years gosherve has grown Prometheus metrics and the ability to serve assets from an embedded filesystem.

As I anticipated, naming the new project was the hardest part. I landed on servy, at least for now. I’m reasonably happy with the code - at the time of writing:

It can serve redirects
It can serve embedded web assets
It provides a metrics server on a separate port
It has reasonable unit/integration test coverage

One of the things I love about Go is the built-in ability to embed static assets into a binary through the //go:embed directive, which gives you a pointer to an embedded filesystem. I was able to achieve a similar effect in Rust with axum-embed, which in turn builds upon rust-embed.

I used Nix to create different variants of the build (i.e. a “vanilla” build, and one that serves my website by creating a derivation just for the web content, and another for the jnsgruk binary).

You can run a copy of this website (frozen in time just before I wrote this post!) by running nix run github:jnsgruk/servy#jnsgruk, or build a container for it by running nix build github:jnsgruk/servy#jnsgruk-container.

I was expecting the Rust binary to be quite a lot smaller. I’m not sure why. The old (Go) binary weighs in at 68MB, where the new Rust binary comes in at 67MB.

So I was right! I guess?! The result here is relatively uninteresting - a good chunk of both binaries is just the static assets (images!) that make up this site. At the time of writing the jnsgruk-content derivation evaluates at around 57MB - meaning there is 10MB and 9MB respectively for Go and Rust added by the actual server code.

Basic Load Testing with `k6` #

I’ve been looking for an excuse to play with k6 for a while. According to the documentation:

Grafana k6 is an open-source, developer-friendly, and extensible load testing tool. k6 allows you to prevent performance issues and proactively improve reliability.

In very simple terms, to use k6 you define a script (in Javascript) that outlines a set of requests to make, their success criteria and (optionally) the strategy for ramping up load on the server. It has many more features than I used for this project, but I was impressed with how simple it was to get started. I began by running the following (grabbing k6 from nixpkgs):

1
2
3
4
5


# Start a shell with k6 available
❯ nix shell github:NixOS/nixpkgs/nixos-unstable#k6
# Init a new k6 script
❯ k6 new
Initialized a new k6 test script in script.js. You can now execute it by running `k6 run script.js`.

Now I just needed to work out what I wanted to test! I wanted a relatively simple test that requested a mix of web assets and redirects for a sustained period, to see how much throughput I could achieve with each of the server implementations. The template came with some sensible setup for the number of VUs (virtual users) and duration of the test:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


import { check } from "k6"
import http from "k6/http"

export const options = {
 // A number specifying the number of VUs to run concurrently.
 vus: 10,
 // A string specifying the total duration of the test run.
 duration: "30s",
}
// ...

VUs are an emulation of a user interacting with your service; each of them is an agent which will execute the test script. Each time the script is executed (by a VU), that’s known as an “iteration”.

I wanted to test a variety of requests, so I first generated a list of valid URLs for my server. I used cariddi which is a web crawler written in Go, combined with jq to create a list of valid paths in a JSON file:

1
2
3
4
5
6
7


# Start a shell with k6 available
❯ nix shell github:NixOS/nixpkgs/nixos-unstable#cariddi
# Run cariddi and generate the paths.json
❯ echo http://localhost:8080 \
 | cariddi -- -plain \
 | cut -d"/" -f4- \
 | jq -r -nR '[inputs | select(length>0)]' > paths.json

Similarly I generated a list of valid redirects:

1
2
3


❯ curl -s "https://gist.githubusercontent.com/jnsgruk/b590f114af1b041eeeab3e7f6e9851b7/raw" \
 | cut -d" " -f1 \
 | jq -r -nR '[inputs | select(length>0)]' > redirects.json

Now in my k6 script, I was able to read those files to create a list of URLs to batch GET during the load test:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


// Allow the host to be overridden by the 'K6_HOST' env var.
const host = __ENV.K6_HOST || "http://localhost:8080"
// Read the list of paths/redirects.
const assets = JSON.parse(open(`./paths.json`))
const redirects = JSON.parse(open(`./redirects.json`))

export default function () {
 // Batch requests to redirects and static assets.
 const responses = http.batch(
 [...assets, ...redirects].map((p) => {
 return ["GET", `${host}/${p}`, null, { redirects: 0 }]
 })
 )

 // For each response, ensure we get a 200/301/308/404
 // response - we shouldn't see anything else.
 responses.forEach((r) => {
 check(r, {
 status: (res) => {
 return [200, 301, 308, 404].includes(res.status)
 },
 })
 })
}

I’m using a pretty naive check function here, though they can be expanded to check for other conditions in the response body, the size of the response, etc.

You can see my final test script on Github.

Finally, I ran k6 and let it rip! This initial run was against servy, running on my workstation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40


❯ k6 run script.js

 /\  Grafana /‾‾/
 /\  / \  |\  __ / /
 / \/ \  | |/ / / ‾‾\
 / \  | ( | (‾) |
 / __________ \  |_|\_\  \_____/

 execution: local
 script: script.js
 output: -

 scenarios: (100.00%) 1 scenario, 10 max VUs, 1m0s max duration (incl. graceful stop):
 * default: 10 looping VUs for 30s (gracefulStop: 30s)

INFO[0030] [k6-reporter v2.3.0] Generating HTML summary report source=console

 ✓ status

 checks.........................: 100.00% ✓ 1331889 ✗ 0
 data_received..................: 148 GB 4.9 GB/s
 data_sent......................: 163 MB 5.4 MB/s
 http_req_blocked...............: avg=2.41µs min=390ns med=1.72µs max=17.3ms p(90)=3.6µs p(95)=4.91µs
 http_req_connecting............: avg=4ns min=0s med=0s max=462.21µs p(90)=0s p(95)=0s
 http_req_duration..............: avg=501.25µs min=42.08µs med=139.77µs max=45.41ms p(90)=291.59µs p(95)=419.65µs
 { expected_response:true }...: avg=501.25µs min=42.08µs med=139.77µs max=45.41ms p(90)=291.59µs p(95)=419.65µs
 http_req_failed................: 0.00% ✓ 0 ✗ 1331889
 http_req_receiving.............: avg=379.39µs min=4.13µs med=29.07µs max=45.23ms p(90)=108.31µs p(95)=177.05µs
 http_req_sending...............: avg=4.45µs min=960ns med=3.48µs max=3.36ms p(90)=6.83µs p(95)=8.79µs
 http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
 http_req_waiting...............: avg=117.4µs min=33.65µs med=98.19µs max=23.03ms p(90)=176.34µs p(95)=225.09µs
 http_reqs......................: 1331889 44333.219428/s
 iteration_duration.............: avg=38.08ms min=4.92ms med=45.46ms max=80.35ms p(90)=47.68ms p(95)=48.53ms
 iterations.....................: 7881 262.326742/s
 vus............................: 10 min=10 max=10
 vus_max........................: 10 min=10 max=10


running (0m30.0s), 00/10 VUs, 7881 complete and 0 interrupted iterations
default ✓ [======================================] 10 VUs 30s

Nice! Between the 10 VUs we configured, k6 managed nearly 8000 iterations, and my new server responded with a total of 148GB of data!

Nix-ified Load Testing #

Now I’d figured out the basics of k6, I wanted to create some infrastructure that would allow me to run load tests in consistent environments against both old and new implementations of my server.

A couple of ideas came to mind here - the first of which was the NixOS integration test driver. I absolutely love this feature of Nix, and the scripts for interacting with the driver are nice and simple to maintain. One slight irritation in this case is that the test machines don’t have access to the internet - which is where the redirects map is fetched from in my precompiled server binaries. It’s certainly possible to get clever with a fake redirects server and some DNS shennanigans in the test machines, but I opted instead to build simple nixosConfigurations which could be started as VMs, similar to the approach taken in a previous post.

I wanted to automate:

The creation of a VM with both the Go and Rust versions of my server
The generation of the paths.json and redirects.json files I created above
Running the k6 load test
Fetching the results from the load test

Defining Test VMs #

I created a new repository with a flake.nix which took nixpkgs, my jnsgr.uk repo and my servy repo as inputs, pinned to the latest revisions at the time of writing:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


{
 description = "web server benchmarking flake";

 inputs = {
 nixpkgs.url = "github:nixos/nixpkgs/nixpkgs-unstable";
 jnsgruk-go.url = "github:jnsgruk/jnsgr.uk/3240a0104c01ae672a6f5f7b0529ad08bcbc8af2";
 jnsgruk-rust.url = "github:jnsgruk/servy/71a337317defc779c6c55d486e20d104c5d478f2";
 };

 # ...
}

Next up I needed a virtual machine definition, which I first defined as a nixosConfiguration in the flake’s outputs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


{
 # ... inputs ...
 outputs = { self, nixpkgs, jnsgruk-go, jnsgruk-rust, ... }:
 let
 forAllSystems = nixpkgs.lib.genAttrs [ "x86_64-linux" "aarch64-linux" ];
 in
 {
 # A minimal NixOS virtual machine which used for testing craft applications.
 nixosConfigurations = forAllSystems (system: {
 benchvm = nixpkgs.lib.nixosSystem {
 specialArgs = {
 inherit self system;
 jnsgruk-go = jnsgruk-go.packages.${system}.jnsgruk;
 jnsgruk-rust = jnsgruk-rust.packages.${system}.jnsgruk;
 };
 modules = [ ./vm.nix ];
 };
 });
 };
}

The actual machine configuration lives in vm.nix. This is a standard NixOS configuration, with the addition of some elements that define the specs of the virtual machine used to boot it (cores, memory, disk, etc.). In this case the machine is configured with just a root user, and the password password. This is not an ideal setup from a security standpoint, but these VMs were only run on my workstation (behind NAT) for a short period of time and the plain text password eased the automation I’m about to describe. A more robust approach would have been to put my SSH public keys into the authorized keys definition for the user, but then you folks wouldn’t have been able to play along as easily!

In this particular configuration, the VM is set up like so:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


virtualisation = {
 forwardPorts = [{
 from = "host";
 host.port = 2222;
 guest.port = 22;
 }];
 cores = 4;
 memorySize = 4096;
 diskSize = 10000;
};

With that in place, we can now boot the VM (in the background):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


❯ nix run .#nixosConfigurations.benchvm.config.system.build.vm -- --daemonize --display none
❯ ssh -p 2222 root@localhost
The authenticity of host '[localhost]:2222 ([127.0.0.1]:2222)' can't be established.
ED25519 key fingerprint is SHA256:icnH3EQAzmjdfCkyPWFljQWaVSCaXdP2M+ekKXd0NlY.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[localhost]:2222' (ED25519) to the list of known hosts.
(root@localhost) Password:

[root@benchvm:~]#

Wrapping `k6` #

Now on to automating the test itself. When configuring the VM, I made sure to include a systemd unit for each of the server implementations, so next I wanted to write a simple script that would execute the load test for each implementation and fetch the results.

Before that, I modified the k6 script to not only output a report to stdout, but also to a text file, a JSON file and a rendered HTML file (I didn’t know at the time which I’d prefer, and I wanted to survey the options!). I achieved this by defining the handleSummary function in the k6 script:

1
2
3
4
5
6
7
8


export function handleSummary(data) {
 return {
 "summary.json": JSON.stringify(data),
 "summary.html": htmlReport(data),
 "summary.txt": textSummary(data, { enableColors: false, indent: "" }),
 stdout: "\n" + textSummary(data, { enableColors: true, indent: "" }) + "\n\n",
 }
}

First I scripted the generation of the paths and the execution of the test script, and packaged the combination up as a Nix package for installation into the test machine. This is just a bash script which ensures only the specified implementation is running (with systemd), generates the list of URLs to test and then runs k6 against the server. That script is packaged as a Nix package called k6test which contains both my bash script and the k6 test script; the Nix package is then added to the VM configuration.

Automating Test Execution #

I borrowed the vm-exec script from a past project, renaming it benchvm-exec, and created a new script called benchvm-test:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


#!/usr/bin/env bash
set -euo pipefail
info() { echo -e "\e[92m[$HOSTNAME] $*\e[0m"; }

# Set some SSH options to ignore host key errors and make logging quieter.
# This is a bad idea in general, but here is used to faciliate comms with
# a brand new VM each time.
SSH_OPTS=(-P 2222 -o "UserKnownHostsFile=/dev/null" -o "StrictHostKeyChecking=no")

info "Running k6test against $1"
benchvm-exec k6test "$1"

info "Collecting results files"
sshpass -ppassword scp "${SSH_OPTS[@]}" root@localhost:summary.json "summary-${1}.json"
sshpass -ppassword scp "${SSH_OPTS[@]}" root@localhost:summary.html "summary-${1}.html"
sshpass -ppassword scp "${SSH_OPTS[@]}" root@localhost:summary.txt "summary-${1}.txt"

info "Results available in summary-${1}.txt, summary-${1}.json and summary-${1}.html"

This was the final piece of the puzzle for now; after building and packaging this script and defining the test VM as a package in the flake, I’d enabled the following workflow:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Start the devShell for the server-bench project
❯ nix develop github:jnsgruk/server-bench
# Build & run the VM
❯ run-benchvm-vm --daemonize --display none
# Start, and load test the gosherve based server
❯ benchvm-test jnsgruk-go
# Start, and load test the Rust based server
❯ benchvm-test jnsgruk-rust
# All done, power down the VM
❯ benchvm-exec poweroff

Excellent! Now on to some actual testing!

Initial Load Test Results #

Having seen the results for servy earlier, I started up the existing gosherve based server and ran the same test, only to discover quite a delta in the results. Where servy managed 4.9 GB/s throughout the test, gosherve only achieved 697 MB/s (sending 21GB in total). The full results are in the details box below, but overall it managed much lower performance across the board on the measurements that k6 makes.

Initial gosherve load-test results

checks.........................: 100.00% ✓ 189449 ✗ 0
data_received..................: 21 GB 697 MB/s
data_sent......................: 26 MB 848 kB/s
http_req_blocked...............: avg=2.02µs min=400ns med=1.42µs max=19.76ms p(90)=2.29µs p(95)=2.77µs
http_req_connecting............: avg=203ns min=0s med=0s max=19.52ms p(90)=0s p(95)=0s
http_req_duration..............: avg=9.3ms min=29.49µs med=9.36ms max=152.39ms p(90)=13.16ms p(95)=14.76ms
{ expected_response:true }...: avg=9.3ms min=29.49µs med=9.36ms max=152.39ms p(90)=13.16ms p(95)=14.76ms
http_req_failed................: 0.00% ✓ 0 ✗ 189449
http_req_receiving.............: avg=106.56µs min=4.88µs med=48.7µs max=6.01ms p(90)=215.06µs p(95)=381.38µs
http_req_sending...............: avg=4.81µs min=950ns med=3.97µs max=19.71ms p(90)=5.93µs p(95)=6.98µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=9.19ms min=22.85µs med=9.26ms max=152.2ms p(90)=12.96ms p(95)=14.51ms
http_reqs......................: 189449 6281.256789/s
iteration_duration.............: avg=268.14ms min=176.88ms med=250.61ms max=433.23ms p(90)=334.14ms p(95)=342.1ms
iterations.....................: 1121 37.1672/s
vus............................: 10 min=10 max=10
vus_max........................: 10 min=10 max=10

I was quite surprised by this: my expectation was that the Go HTTP server would outperform what I’d put together in Rust. I decided to look a little deeper and see if I could figure out why, or at least why the delta was so big.

Profiling `gosherve` with Parca #

For last couple of years, I’ve been playing with Parca, which is a continuous profiling tool written in Go by the folks at Polar Signals. I wrote about it back in 2022 on Charmhub, demonstrating how Parca could be used for profiling applications deployed with Juju.

Instrumenting `gosherve` #

There are two components to the open source Parca offering - the server backend, and a zero-instrumentation, eBPF-based agent which can be used for on-CPU profiling of any workload. Because gosherve is Go based, I didn’t need the agent so long as I enabled the pprof endpoint in my server, which is trivial for almost any Go application. I took main.go and applied the following changes to ensure that I could hit the pprof endpoints on port 6060:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


diff --git a/main.go b/main.go
index 964ceb2..374c440 100644
--- a/main.go
+++ b/main.go
@@ -10,6 +10,9 @@ import (
 "log/slog"
 "os"

+ "net/http"
+ _ "net/http/pprof"
+
 "github.com/jnsgruk/gosherve/pkg/logging"
 "github.com/jnsgruk/gosherve/pkg/server"
 )
@@ -29,6 +32,11 @@ func main() {
 flag.Parse()
 logging.SetupLogger(*logLevel)

+ go func() {
+ http.ListenAndServe(":6060", nil)
+ slog.Info("pprof server started on port 6060")
+ }()
+
 // Create an fs.FS from the embedded filesystem
 fsys, err := fs.Sub(publicFS, "public")
 if err != nil {

Next I fired up Parca (having first packaged it for NixOS 😉) and configured it to scrape the new jnsgruk binary while I ran the same test against it, which resulted in the following profile (which you can explore yourself on pprof.me):

Looking at this, we can see that around a third of the time was spent in garbage collection. This seemed high; it was more prominent on this run than others but it was persistently a high portion of the CPU time. Something to loop back to! However, inside the route handler itself there are two things that stood out to me:

Initial optimisations #

In the red box, you can see about a third of the time spent in the route handler was in creating SHA1 sums, which is happening here to calculate the ETag for content before serving it. This could probably be optimised - either by changing how the ETag is calculated (i.e. not hashing the whole file contents), or by caching ETags in memory, particularly given that in the case of my website, gosherve only serves assets which are embedded (read only) in the binary, so the ETag will never change for a given file for a particular compiled binary.

In the blue box, we can see that another third of the time spent is on refreshing the redirects. This seemed strange; the redirects don’t change throughout the life of the test, and we’re only requesting known redirects from our redirects.json file. This seemed much more likely to be a culprit in the poor load test results, because unlike calculating a SHA1 sum (which is almost entirely CPU-bound), this function is making an outbound (TLS) connection to Github and potentially blocking on IO before parsing the response, rechecking the redirects map for a match, etc.

As I looked at the code, I noticed an obvious (and somewhat embarrassing!) mistake I’d made when implementing gosherve. Because the server tries to parse redirects before looking for files with a matching path, every request for a file causes the redirects map to be updated from its upstream source 🤦. Looking back at the new servy implementation, I hadn’t made the same mistake - perhaps this was the reason the Rust version was so much faster?

I made a small change to gosherve and re-ran the test, which resulted in this profile and quite a substantial bump in request performance!

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


checks.........................: 100.00% ✓ 913783 ✗ 0
data_received..................: 101 GB 3.4 GB/s
data_sent......................: 123 MB 4.1 MB/s
http_req_blocked...............: avg=3.21µs min=390ns med=2.12µs max=14.12ms p(90)=4.19µs p(95)=5.7µs
http_req_connecting............: avg=70ns min=0s med=0s max=14.09ms p(90)=0s p(95)=0s
http_req_duration..............: avg=1.8ms min=42.12µs med=899.13µs max=31.31ms p(90)=4.86ms p(95)=6.45ms
 { expected_response:true }...: avg=1.8ms min=42.12µs med=899.13µs max=31.31ms p(90)=4.86ms p(95)=6.45ms
http_req_failed................: 0.00% ✓ 0 ✗ 913783
http_req_receiving.............: avg=261.88µs min=4.85µs med=63.59µs max=17.8ms p(90)=620.15µs p(95)=1.31ms
http_req_sending...............: avg=8.21µs min=1.05µs med=5.21µs max=8.73ms p(90)=9.87µs p(95)=13.07µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=1.53ms min=30.81µs med=656.71µs max=30.76ms p(90)=4.38ms p(95)=5.91ms
http_reqs......................: 913783 30435.133157/s
iteration_duration.............: avg=55.5ms min=28.86ms med=55.18ms max=77.07ms p(90)=61.02ms p(95)=62.98ms
iterations.....................: 5407 180.089545/s
vus............................: 10 min=10 max=10
vus_max........................: 10 min=10 max=10

The overall throughput increased by five times, with a decrease in latency across the board. Still around a third less overall throughput than the Rust server, but a huge improvement nonetheless.

I subsequently experimented with (naively) removing the ETag calculation to see whether or not it was worth implementing some caching - but it actually resulted in very little difference in throughput and CPU utilisation. Besides, while the calculation takes some CPU time, in a real-life deployment it reduces the overall load on the server by ensuring that http.ServeContent only sends actual content when there is a new version, relying more heavily on the user’s browser cache. My k6 tests (deliberately) weren’t sending the right headers (such as If-None-Match) to benefit from the behaviour one might expect from a browser, and it was just requesting the same files over and over again with the equivalent of a cold/empty cache.

Reducing Allocations #

Now back to all those allocations, and the time spent in garbage collection as a result… gosherve was spending about a third of it’s time in the io.ReadFile function, and given the amount of time spent in garbage collection, any optimisation made to the number of allocations on the heap would likely yield another big performance increase.

The problem lied in the routeHandler, where I was reading the entire contents of each file that was being served into memory. fs.ReadFile makes an allocation the size of the file it’s reading, meaning the contents of the file end up on the heap. In a situation like our load test - this means the entire contents of my website ended up on the heap, and the Go runtime was busily garbage collecting to clean up.

I looked around at alternative ways to implement the same functionality in a more efficient manner. In Go 1.22, http.ServeFileFS was introduced as a counterpart to http.ServeFile. http.ServeFileFS operates on an fs.FS rather than the host filesystem. Under the hood, rather than reading the whole file into memory ServeFileFS is using io.Copy and thus io.copyBuffer which allocates a single buffer that fits on the stack - fewer allocations means less memory usage, and less time spent occupying cores with garbage collection!

The change to gosherve was pretty simple - you can see a slightly shortened diff below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


diff --git a/pkg/server/route_handler.go b/pkg/server/route_handler.go
index 2417e45..e0c9d68 100644
--- a/pkg/server/route_handler.go
+++ b/pkg/server/route_handler.go
@@ -1,7 +1,6 @@
 // snip!

- // Try reading the file and return early if that fails
- b, err := fs.ReadFile(*s.webroot, filepath)
- if err != nil {
- return false
- }
-
 w.Header().Set("Cache-Control", "public, max-age=31536000, must-revalidate")
- w.Header().Set("ETag", fmt.Sprintf(`"%d-%x"`, len(b), sha1.Sum(b)))
+ w.Header().Set("ETag", fmt.Sprintf(`"%s-%d-%x"`, fi.Name(), fi.Size(), sha1.Sum([]byte(fi.ModTime().String()))))

- http.ServeContent(w, r, filepath, time.Now(), bytes.NewReader(b))
+ http.ServeFileFS(w, r, *s.webroot, filepath)
 s.metrics.responseStatus.WithLabelValues(strconv.Itoa(http.StatusOK)).Inc()
 l.Info("served file", slog.Group("response", "status_code", http.StatusOK, "file", filepath))

This change also necessitated a change to ETag calculation, which now only hashes the filename, size and modified time. There is a slight oddity in that files from an embedded filesystem have their timestamps all fixed (to avoid changes in static files messing with build reproducibility), but since the files cannot change during a binary’s lifetime, the ETag calculated from the modified timestamp is still just as valid.

I quickly made a new build of jnsgruk using a locally referenced copy of gosherve and ran k6/parca to check the outcome. You can explore the profile yourself on pprof.me:

Here you can clearly see that the runtime is spending much less time in garbage collection, and the call to io.Copy/io.copyBuffer without the calls to fs.ReadFile. It’s now also much harder to spot the ETag generation - likely because we’re now only hashing a small string, rather than the whole file contents.

The k6 results came back as follows 🚀:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


checks.........................: 100.00% ✓ 1525300 ✗ 0
data_received..................: 170 GB 5.7 GB/s
data_sent......................: 207 MB 6.9 MB/s
http_req_blocked...............: avg=4µs min=410ns med=2.36µs max=18.47ms p(90)=5.02µs p(95)=6.77µs
http_req_connecting............: avg=40ns min=0s med=0s max=17.98ms p(90)=0s p(95)=0s
http_req_duration..............: avg=999.6µs min=42.17µs med=645.03µs max=24.22ms p(90)=2.34ms p(95)=3.09ms
 { expected_response:true }...: avg=999.6µs min=42.17µs med=645.03µs max=24.22ms p(90)=2.34ms p(95)=3.09ms
http_req_failed................: 0.00% ✓ 0 ✗ 1525300
http_req_receiving.............: avg=245.43µs min=4.64µs med=75.29µs max=23.48ms p(90)=626.08µs p(95)=1.18ms
http_req_sending...............: avg=9.69µs min=1µs med=5.49µs max=19.54ms p(90)=10.86µs p(95)=14.46µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=744.47µs min=28.09µs med=395.81µs max=20.62ms p(90)=1.89ms p(95)=2.58ms
http_reqs......................: 1525300 50823.211595/s
iteration_duration.............: avg=34.41ms min=11.91ms med=34.44ms max=67.75ms p(90)=37.6ms p(95)=38.61ms
iterations.....................: 8716 290.418352/s
vus............................: 10 min=10 max=10
vus_max........................: 10 min=10 max=10

The Go server is now in the lead! There isn’t a huge margin here between the Rust server and the Go server, but this is about the sort of difference I was expecting based on the reading I’d done before doing the testing.

Profiling `servy`? #

I did profile servy using parca-agent, but I’ve yet to look at the results in detail. The profile is a lot more complex (see below), partially as a result of how much work is going on under the hood when standing up an axum server with hyper, tower and tokio. I’m going to spend some more time with this profile over the coming weeks, but you can explore it for yourself on pprof.me.

If you do play with the profile on pprof.me, try filtering by function name and entering servy, and you’ll see the parts of the code I wrote highlighted on the profile to explore:

By the time you read this, the parca-agent should be upstream into nixpkgs, but in the mean time you can pull the package I included in the server-bench repo and use that:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Start the devShell for the server-bench project
❯ nix develop github:jnsgruk/server-bench
# Start the agent
❯ sudo parca-agent -- \
 --remote-store-address="grpc.polarsignals.com:443" \
 --remote-store-bearer-token="$TOKEN" \
 --node="$HOSTNAME"

# Or, on Ubuntu (see polarsignals.com/docs/setup-collection-snaps)
❯ sudo snap install --classic parca-agent

In my Nix example above, I’m sending profiles to Polar Signals Cloud, but you can also set the remote-store-address to the address of a locally running parca instance no problem.

Depending on the language you’re profiling, you’ll either need to include debug info in the binary you’re profiling (which is what I did), or upload your source/debug info manually. There are more instructions for that in the docs.

Final Results & Comparisons #

I wanted to gather some test results from all three targets across a number of different machine configurations; my website runs on a teeny-tiny Fly.io instance with 1 vCPU and 256MB RAM, so I was interested to see how performance changed on machines with varying configurations. I’ve included a summary in the table below which highlights just the data_received measurement from each test, and you can see the full reports from k6 by expanding the details box:

Expand for full `k6` reports

1 vCPU/256MB RAM/1 VUs - gosherve (old)

checks.........................: 100.00% ✓ 23322 ✗ 0
data_received..................: 2.6 GB 86 MB/s
data_sent......................: 3.1 MB 105 kB/s
http_req_blocked...............: avg=2.41µs min=400ns med=1.28µs max=1.04ms p(90)=1.87µs p(95)=2.27µs
http_req_connecting............: avg=50ns min=0s med=0s max=508.13µs p(90)=0s p(95)=0s
http_req_duration..............: avg=7.57ms min=49.54µs med=7.71ms max=143.13ms p(90)=9.42ms p(95)=10.65ms
{ expected_response:true }...: avg=7.57ms min=49.54µs med=7.71ms max=143.13ms p(90)=9.42ms p(95)=10.65ms
http_req_failed................: 0.00% ✓ 0 ✗ 23322
http_req_receiving.............: avg=98.03µs min=4.46µs med=57.25µs max=7.69ms p(90)=177.19µs p(95)=250.6µs
http_req_sending...............: avg=5.7µs min=970ns med=3.1µs max=480.07µs p(90)=4.79µs p(95)=9.26µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=7.47ms min=37.26µs med=7.62ms max=142.9ms p(90)=9.24ms p(95)=10.33ms
http_reqs......................: 23322 776.881114/s
iteration_duration.............: avg=217.52ms min=187.06ms med=198.66ms max=359.52ms p(90)=276.89ms p(95)=280.19ms
iterations.....................: 138 4.59693/s
vus............................: 1 min=1 max=1
vus_max........................: 1 min=1 max=1

1 vCPU/256MB RAM/1 VUs - gosherve

checks.........................: 100.00% ✓ 158691 ✗ 0
data_received..................: 18 GB 587 MB/s
data_sent......................: 21 MB 714 kB/s
http_req_blocked...............: avg=1.89µs min=379ns med=1.08µs max=1.15ms p(90)=1.62µs p(95)=1.98µs
http_req_connecting............: avg=11ns min=0s med=0s max=655.05µs p(90)=0s p(95)=0s
http_req_duration..............: avg=1.04ms min=48.25µs med=537.57µs max=31.33ms p(90)=2.63ms p(95)=3.66ms
{ expected_response:true }...: avg=1.04ms min=48.25µs med=537.57µs max=31.33ms p(90)=2.63ms p(95)=3.66ms
http_req_failed................: 0.00% ✓ 0 ✗ 158691
http_req_receiving.............: avg=91.12µs min=3.66µs med=20.8µs max=23.05ms p(90)=156.29µs p(95)=287.17µs
http_req_sending...............: avg=6.47µs min=900ns med=2.57µs max=2.22ms p(90)=4.06µs p(95)=6.21µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=949.02µs min=35.88µs med=466.14µs max=31.3ms p(90)=2.46ms p(95)=3.47ms
http_reqs......................: 158691 5287.057691/s
iteration_duration.............: avg=31.95ms min=21.16ms med=30.39ms max=107.46ms p(90)=36.19ms p(95)=41.18ms
iterations.....................: 939 31.284365/s
vus............................: 1 min=1 max=1
vus_max........................: 1 min=1 max=1

1 vCPU/256MB RAM/1 VUs - servy

checks.........................: 100.00% ✓ 148213 ✗ 0
data_received..................: 16 GB 547 MB/s
data_sent......................: 18 MB 605 kB/s
http_req_blocked...............: avg=1.5µs min=350ns med=971ns max=1.11ms p(90)=1.48µs p(95)=1.82µs
http_req_connecting............: avg=11ns min=0s med=0s max=451.97µs p(90)=0s p(95)=0s
http_req_duration..............: avg=928.79µs min=69.84µs med=671.38µs max=55.59ms p(90)=1.57ms p(95)=2.18ms
{ expected_response:true }...: avg=928.79µs min=69.84µs med=671.38µs max=55.59ms p(90)=1.57ms p(95)=2.18ms
http_req_failed................: 0.00% ✓ 0 ✗ 148213
http_req_receiving.............: avg=136.98µs min=3.76µs med=17.14µs max=46.31ms p(90)=103.7µs p(95)=192.52µs
http_req_sending...............: avg=5.33µs min=890ns med=2.35µs max=2.99ms p(90)=3.69µs p(95)=5.05µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=786.47µs min=55.89µs med=629.47µs max=17.92ms p(90)=1.43ms p(95)=1.96ms
http_reqs......................: 148213 4940.080272/s
iteration_duration.............: avg=34.19ms min=16.57ms med=25.68ms max=109.45ms p(90)=55.08ms p(95)=57.83ms
iterations.....................: 877 29.231244/s
vus............................: 1 min=1 max=1
vus_max........................: 1 min=1 max=1

2 vCPU/4GB RAM/10 VUs - gosherve (old)

checks.........................: 100.00% ✓ 116272 ✗ 0
data_received..................: 13 GB 426 MB/s
data_sent......................: 16 MB 518 kB/s
http_req_blocked...............: avg=2.25µs min=380ns med=1.06µs max=4.3ms p(90)=1.69µs p(95)=2.05µs
http_req_connecting............: avg=475ns min=0s med=0s max=4.27ms p(90)=0s p(95)=0s
http_req_duration..............: avg=15.37ms min=29.8µs med=14.28ms max=145.82ms p(90)=24.79ms p(95)=29.78ms
{ expected_response:true }...: avg=15.37ms min=29.8µs med=14.28ms max=145.82ms p(90)=24.79ms p(95)=29.78ms
http_req_failed................: 0.00% ✓ 0 ✗ 116272
http_req_receiving.............: avg=174.41µs min=4.15µs med=28.84µs max=21.95ms p(90)=241.72µs p(95)=703.97µs
http_req_sending...............: avg=7.22µs min=890ns med=2.98µs max=5.58ms p(90)=5.8µs p(95)=9.15µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=15.19ms min=22.98µs med=14.11ms max=144.5ms p(90)=24.48ms p(95)=29.37ms
http_reqs......................: 116272 3837.442876/s
iteration_duration.............: avg=439.41ms min=323.27ms med=430.04ms max=567.74ms p(90)=505.87ms p(95)=516.96ms
iterations.....................: 688 22.706763/s
vus............................: 10 min=10 max=10
vus_max........................: 10 min=10 max=10

2 vCPU/4GB RAM/10 VUs - gosherve

checks.........................: 100.00% ✓ 362167 ✗ 0
data_received..................: 40 GB 1.3 GB/s
data_sent......................: 49 MB 1.6 MB/s
http_req_blocked...............: avg=2.23µs min=360ns med=770ns max=12.31ms p(90)=1.34µs p(95)=1.66µs
http_req_connecting............: avg=117ns min=0s med=0s max=1.84ms p(90)=0s p(95)=0s
http_req_duration..............: avg=4.7ms min=32.38µs med=3.94ms max=47.81ms p(90)=9.04ms p(95)=11.63ms
{ expected_response:true }...: avg=4.7ms min=32.38µs med=3.94ms max=47.81ms p(90)=9.04ms p(95)=11.63ms
http_req_failed................: 0.00% ✓ 0 ✗ 362167
http_req_receiving.............: avg=196.28µs min=3.38µs med=15.42µs max=25.62ms p(90)=142.41µs p(95)=896.21µs
http_req_sending...............: avg=8.44µs min=890ns med=2.19µs max=18.13ms p(90)=3.63µs p(95)=6.34µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=4.5ms min=25.57µs med=3.77ms max=45.46ms p(90)=8.74ms p(95)=11.13ms
http_reqs......................: 362167 12040.737924/s
iteration_duration.............: avg=140.21ms min=80.54ms med=139.82ms max=221.91ms p(90)=153.25ms p(95)=157.95ms
iterations.....................: 2143 71.24697/s
vus............................: 10 min=10 max=10
vus_max........................: 10 min=10 max=10

2 vCPU/4GB RAM/10 VUs - servy

checks.........................: 100.00% ✓ 464412 ✗ 0
data_received..................: 51 GB 1.7 GB/s
data_sent......................: 57 MB 1.9 MB/s
http_req_blocked...............: avg=1.99µs min=360ns med=720ns max=14.38ms p(90)=1.13µs p(95)=1.42µs
http_req_connecting............: avg=92ns min=0s med=0s max=6.71ms p(90)=0s p(95)=0s
http_req_duration..............: avg=3.67ms min=44.91µs med=3.07ms max=60.17ms p(90)=5.9ms p(95)=7.98ms
{ expected_response:true }...: avg=3.67ms min=44.91µs med=3.07ms max=60.17ms p(90)=5.9ms p(95)=7.98ms
http_req_failed................: 0.00% ✓ 0 ✗ 464412
http_req_receiving.............: avg=206.58µs min=3.41µs med=14.06µs max=56.96ms p(90)=56.97µs p(95)=143.43µs
http_req_sending...............: avg=7.12µs min=870ns med=1.62µs max=14.84ms p(90)=3.01µs p(95)=5.78µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=3.45ms min=35.61µs med=3.01ms max=24.64ms p(90)=5.7ms p(95)=7.54ms
http_reqs......................: 464412 15448.582291/s
iteration_duration.............: avg=109.3ms min=63.59ms med=108.88ms max=161.77ms p(90)=120.71ms p(95)=124.65ms
iterations.....................: 2748 91.41173/s
vus............................: 10 min=10 max=10
vus_max........................: 10 min=10 max=10

16 vCPU/16GB RAM/10 VUs - gosherve (old)

checks.........................: 100.00% ✓ 119990 ✗ 0
data_received..................: 13 GB 439 MB/s
data_sent......................: 16 MB 534 kB/s
http_req_blocked...............: avg=2.05µs min=440ns med=1.55µs max=972.62µs p(90)=2.32µs p(95)=2.74µs
http_req_connecting............: avg=38ns min=0s med=0s max=275.54µs p(90)=0s p(95)=0s
http_req_duration..............: avg=14.82ms min=35.49µs med=15.16ms max=148.26ms p(90)=21.12ms p(95)=23.89ms
{ expected_response:true }...: avg=14.82ms min=35.49µs med=15.16ms max=148.26ms p(90)=21.12ms p(95)=23.89ms
http_req_failed................: 0.00% ✓ 0 ✗ 119990
http_req_receiving.............: avg=104.7µs min=5.07µs med=43.71µs max=8.96ms p(90)=228.22µs p(95)=396.05µs
http_req_sending...............: avg=4.52µs min=940ns med=3.99µs max=944.81µs p(90)=6.28µs p(95)=7.86µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=14.71ms min=23.97µs med=15.05ms max=148.25ms p(90)=20.95ms p(95)=23.7ms
http_reqs......................: 119990 3957.203279/s
iteration_duration.............: avg=426.57ms min=343.33ms med=422.38ms max=526.97ms p(90)=491ms p(95)=502.47ms
iterations.....................: 710 23.415404/s
vus............................: 10 min=10 max=10
vus_max........................: 10 min=10 max=10

16 vCPU/16GB RAM/10 VUs - gosherve

checks.........................: 100.00% ✓ 1297244 ✗ 0
data_received..................: 144 GB 4.8 GB/s
data_sent......................: 175 MB 5.8 MB/s
http_req_blocked...............: avg=2.83µs min=419ns med=1.63µs max=7.97ms p(90)=2.95µs p(95)=3.78µs
http_req_connecting............: avg=2ns min=0s med=0s max=142.69µs p(90)=0s p(95)=0s
http_req_duration..............: avg=1.22ms min=31.96µs med=697.97µs max=36.9ms p(90)=3.16ms p(95)=4.07ms
{ expected_response:true }...: avg=1.22ms min=31.96µs med=697.97µs max=36.9ms p(90)=3.16ms p(95)=4.07ms
http_req_failed................: 0.00% ✓ 0 ✗ 1297244
http_req_receiving.............: avg=234.98µs min=4.21µs med=39.77µs max=35.6ms p(90)=640.71µs p(95)=1.24ms
http_req_sending...............: avg=7.19µs min=930ns med=3.91µs max=30.11ms p(90)=6.84µs p(95)=9.22µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=980.01µs min=19.74µs med=465.75µs max=16.64ms p(90)=2.75ms p(95)=3.68ms
http_reqs......................: 1297244 43221.02723/s
iteration_duration.............: avg=39.08ms min=23.12ms med=38.84ms max=76ms p(90)=43.87ms p(95)=45.71ms
iterations.....................: 7676 255.745723/s
vus............................: 10 min=10 max=10
vus_max........................: 10 min=10 max=10

16 vCPU/16GB RAM/10 VUs - servy

checks.........................: 100.00% ✓ 1407770 ✗ 0
data_received..................: 156 GB 5.2 GB/s
data_sent......................: 172 MB 5.7 MB/s
http_req_blocked...............: avg=2.28µs min=420ns med=1.64µs max=3.96ms p(90)=3µs p(95)=3.94µs
http_req_connecting............: avg=2ns min=0s med=0s max=292.26µs p(90)=0s p(95)=0s
http_req_duration..............: avg=512.52µs min=40.21µs med=177.39µs max=45.65ms p(90)=444.41µs p(95)=623.84µs
{ expected_response:true }...: avg=512.52µs min=40.21µs med=177.39µs max=45.65ms p(90)=444.41µs p(95)=623.84µs
http_req_failed................: 0.00% ✓ 0 ✗ 1407770
http_req_receiving.............: avg=324.43µs min=4.02µs med=26.56µs max=45.31ms p(90)=88.04µs p(95)=147.94µs
http_req_sending...............: avg=4.33µs min=969ns med=3.35µs max=4.11ms p(90)=5.86µs p(95)=7.31µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=183.74µs min=31.45µs med=132.09µs max=8.96ms p(90)=344.35µs p(95)=465.04µs
http_reqs......................: 1407770 46854.747614/s
iteration_duration.............: avg=36.03ms min=5.19ms med=45.31ms max=77.29ms p(90)=48.06ms p(95)=48.95ms
iterations.....................: 8330 277.247027/s
vus............................: 10 min=10 max=10
vus_max........................: 10 min=10 max=10

	`gosherve (old)`	`gosherve`	`servy`
1 vCPU/256MB RAM/1 VUs	86MB/s	587MB/s	547MB/s
2 vCPU/4GB RAM/10 VUs	426MB/s	1.3GB/s	1.7GB/s
16 vCPU/16GB RAM/10 VUs	439MB/s	4.8GB/s	5.2GB/s

Interestingly, it seems that the servy variant actually outperformed gosherve again in these VM-based tests, where it didn’t when run directly on my machine. In any case, the gap is now much smaller between the two, and gosherve still seems to maintain an edge on very small machines.

Note I had to drop the VUs to 1 on the smallest iteration, or I was seeing all three of the variants getting OOM-killed!

I also added a new flake input in the server-bench repo that adds the gosherve based repo again, but pinned to the pre-optimised version.

As such, if you want to try reproducing any of these results and measure old vs. new vs. servy, you just need to do the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Start the devShell for the server-bench project
❯ nix develop github:jnsgruk/server-bench
# Build & run the VM
❯ run-benchvm-vm --daemonize --display none
# Edit the core/memory count as required
❯ vim vm.nix
# Start, and load test a server, choosing your variety
❯ benchvm-test <jnsgruk-go-old|jnsgruk-go|jnsgruk-rust>
# Power down the VM
❯ benchvm-exec poweroff

The results will be gathered in a collection of summary-*.<txt|html|json> files for you to inspect.

Summary #

This was supposed to be a blog about learning Rust, but I’m pleased with where it ended up! This was by no means a particularly in-depth dive into measuring the performance of web servers, focusing almost entirely on the request throughput. There are other factors that could be taken into consideration - such as the resulting memory pressure on the machine, CPU usage, etc. I may come back to this in another post with more comprehensive measurement of both.

Hopefully this post illustrated the power of load testing and profiling tools - even when only used at quite a surface level, and demonstrated how Nix can be used to create robust structures around existing tools to help with such workflows.

I owe a thank you to Frederic from Polar Signals, who gave me a bunch of helpful tips along the way while I was using Parca and profiling gosherve. After my initial improvement in the request handler, Frederic helped me track down and remove the use of fs.ReadFile and further improve the performance of gosherve.

I’m still very early in my Rust development, so if you’ve spotted something horrible in servy, or you’ve got experience with k6 and parca, then reach out and let me know! I’d love to see the creative ways people use these tools.

The performance gap between servy and gosherve is now quite narrow. Close enough, in fact, that they seem to win over one another depending on different conditions in the test environment, and never by a particularly high margin. I haven’t migrated this site over yet, but perhaps I will in the near future. Besides, Fly.io are already kindly limiting the damage that can be done by too many requests for me!

See you next time!

Hot Tub Monitoring with Home Assistant and ESPHome

Tue, 12 Nov 2024 00:00:00 +0000

Introduction #

This year we put a wood-fired hot tub in our back garden. It’s a total indulgence and a real treat. I originally purchased the tub for my wife, but I must admit to having grown quite fond of it myself!

The tub itself is made from Canadian Redwood Cedar. The photo below was taken shortly after it was assembled, so there is still a little bit of leakage which stopped a few days later once the wood had expanded. The 30kW wood burner is the most effective way to heat the tub (which holds around 1700l of water), but there is also a small electric heater in line with the pump & filter which are stashed behind the tub.

There is something wonderfully simple and low-tech about the whole arrangement, but naturally I wanted to get an understanding of the energy usage, and get a reading on the temperature so that I could more accurately set it up for when we wanted to use it. Part of the reasoning for understanding the energy usage was to ensure that the pump and UV filter only fire up when the house is generating enough solar energy to cover it.

A picture paints a thousand words, so you can see what I’m talking about below:

Energy Usage Monitoring #

Starting with the easy part: monitoring the energy usage for each of the components. What you can’t see in the picture above is the following:

An Elecro Vulcan 3kw heater
An evoUV 15w UV clarifier
A Crystal Enterprises CC2513 pump
A Crystal Enterprises CC3030 filter unit

The UV filter, heater and sand filter are all essentially “in line” - when the pump is switched on, water is pulled from the bottom of the tub, through the UV clarifier, sand filter and heater, and then back into the tub (not necessarily in that order!). The heater has a thermostat in it so that it can cut in/out depending on the desired temperature as the water passes through.

In reality, the 3kw heater would take 12-15 hours to heat the tub from scratch, which is why we use the burner that usually takes around 1.5-2 hours. The electric heater is good for maintaining the temperature once the initial heating has been done.

I’ve had quite a lot of success with TP-Link Tapo P110 smart plugs. They’re very cheap, and there is already a competent Home Assistant integration for them. The only downside is that they require a proprietary app for initial setup. After having bought a few of these, I learned about Tasmota, and will probably buy smart plugs compatible with this or similar open source firmwares in the future.

My friends from the Self Hosted Podcast seem to recommend buying pre-flashed smart devices from cloudfree.shop, which I’ll certainly consider next time.

All of the above said - the TP-Link plugs work great for this use-case. I’ve got one for the pump and one for the heater. The pump is configured to run for 2 hours a day during peak daylight hours, and the heater can be toggled as and when I need it. The result, when added to Home Assistant is some neat energy monitoring and the ability to toggle things on and off easily when I need:

In an alternative view that summarises power usage for my home, there is a neat section covering the usage from individual devices (all connected to smart plugs):

As you can see - the heater is incredibly power-hungry, and as a result we hardly use it. In the two hours it ran on the day shown above, it consumed almost twice the energy that my main workstation used all day. We now constrain it’s use to a couple of hours per day in the summer months when solar generation is at its peak - otherwise we just heat the tub with the wood burner as we want.

Temperature Sensor: Initial Solution #

When the tub originally arrived, I looked around for an internet-connected pool thermometer, and was quite surprised at the lack of options. I had assumed this was something relatively common. I ended up ordering an Inkbird IBS-P02R kit.

The kit comprises a small floating thermometer, and a display unit that communicates wirelessly with the thermometer. I picked this particular model because the display unit can be connected to Wi-Fi and then checked on through an app. I had (naively…) assumed that I’d be able to get access through some sort of API, but that turned out not to be the case.

Overall, this solution worked okay. The temperature is reported in 5/10/15 minute intervals according to the configuration, and I had no issues with range/connectivity even though the tub is some distance from my house (and the house is heavily insulated with triple glazing!).

The Inkbird app for iOS leaves a lot to be desired, though. I found the user experience pretty frustrating. The web service was often slow to respond, and it required an annoying number of clicks to get to the information I wanted.

While researching the different ways I could get access to the information, I re-discovered the Inkbird IBS-P01B which is a very similar thermometer to the one I’d bought, but communicates over bluetooth. I’d initially ruled this out since my server is well out of bluetooth range, but I’d also been looking for an excuse to play with an ESP32-based microcontroller…

Temperature Sensor: Homemade Solution #

After a little bit of research I ended up ordering the following:

1x Inkbird IBS-P01B thermometer
1x ESP32-WROOM-32U development board

The former was ordered from AliExpress, and the latter from Amazon. I chose the ESP32-WROOM-32U specifically because it has an external antenna. Given the eventual placement of the ESP32 and the location of the hot tub, I wanted to maximise the chances of establishing a good bluetooth connection.

This was to be my first foray into ESP32/Arduino type development. I’d read (and heard…) lots about ESPHome, and given my goal was to integrate with Home Assistant, this felt like the right route.

Connecting the ESP32 #

The first step was to flash ESPHome onto the device. The development board I bought comes with a CP2012 USB to UART bridge on the board, so connecting the device to my machine was as simple as plugging it in with a MicroUSB cable.

Before the device can be flashed, it needs to be put into programming mode. In my case, the board has a handy BOOT button. The process for entering programming mode was therefore (starting with the device unplugged and powered down):

Press and hold the BOOT button
Connect the ESP32 over USB
Wait a few seconds
Release the BOOT button

Other development boards may not have a button, but all it really does is bridge GPIO0 and GND on the board which can be done with a wire, too. There are good docs on the connection process on the ESPHome website.

ESPHome Support #

ESPHome doesn’t natively support the IBS-P01B, but after some reading it seemed that the underlying messaging format/protocol is similar enough to the IBS-TH1 and IBS-TH2 that the same configuration can be used.

I also came across this blog post where the author had manually created an ESPHome configuration that appeared to work, and with the exact same hardware.

Nonetheless - to my mind, simpler is better most of the time, so I set about creating a simple configuration using the supported ESPHome platform.

ESPHome Dashboard #

There are a few different ways to get started with ESPHome. You can “install” using Docker, which seems to be the preferred method (passing through /dev/ttyUSBx to the container for USB connectivity), or manually using Python packages.

I tried to follow the instructions to get started with Home Assistant, but it seems this is not compatible with “unsupervised” Home Assistant servers like mine.

The esphome package is readily available in nixpkgs, so I was able to get started and fire up the dashboard like so:

❯ nix run unstable#esphome -- dashboard .
2024-11-12 14:47:19,072 INFO Starting dashboard web server on http://0.0.0.0:6052 and configuration dir ....

Browsing to http://localhost:6052 yielded me the following page:

Creating a Firmware #

I proceeded to create a new device, and entered some basic information about my Wifi network:

This is where things got interesting (for me, at least!). It turns out that the ESPHome dashboard makes use of the Web Serial API to program the ESP32 chip over USB from the browser. Given that my board was already connected and in programming mode, this was pretty simple!

The first time I tried this, I got a permission error. After looking through the messages in my kernel’s ring buffer with journalctl -k, and looking at the permissions on the /dev/ttyUSB0 device, it seemed likely that the issue was the root:root ownership with limited permissions. I solved this by running the following:

1
2


sudo chown root:users /dev/ttyUSB0
sudo chmod g+rwx /dev/ttyUSB0

This changed the group of the serial device to the same group as my user, then gave the group read, write and execute permissions. I retried in the browser and the device was flashed with the new firmware, connecting it to my network! Once the device was online, it appeared as such in the ESPHome dashboard, and I cooked up the following configuration by studing the docs on the inkbird_ibsth1_mini sensor component:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44


substitutions:
 name: esphome-web-caf3b8
 friendly_name: Tub Monitor

esphome:
 name: ${name}
 friendly_name: ${friendly_name}
 min_version: 2024.6.0
 name_add_mac_suffix: false
 project:
 name: esphome.web
 version: dev

esp32:
 board: esp32dev
 framework:
 type: arduino

web_server:
improv_serial:
logger:
api:
ota:
 - platform: esphome

wifi:
 # Pulled in from a separate secrets.yaml
 ssid: !secret wifi_ssid
 password: !secret wifi_password

esp32_ble_tracker:
 scan_parameters:
 active: true
 continuous: true
 duration: 1min

sensor:
 - platform: inkbird_ibsth1_mini
 # Bluetooth MAC address of my IBS-P01B
 mac_address: <redacted>
 temperature:
 name: "Tub Temperature"
 battery_level:
 name: "Tub Monitor Battery"

A few points on the above:

The device setup wizard created a secrets.yaml which can hold secrets to be referenced with the !secret <secret name> syntax. This is the same format as used by Home Assistant.
Specifying web_server: enables a small embedded web server that can be used to check status (see below!).
I had to specify the MAC address of the pool thermometer, which I collected using the Bluetooth Inspector app for iOS.

I clicked Install in the top right of the window, and the firmware was compiled and flashed to the device! This time I was able to flash over Wifi since the device was now connected to my network as a result of the onboarding process:

After selecting “Wirelessly”, and waiting for the firmware to be flashed, the logs shortly started flowing in:

To double check things were working, I then browsed to the IP address of the ESPHome device, and was greeted with a simple page that shows the current state of the device:

Adding to Home Assistant was as simple as adding a new integration of type ESPHome and specifying the IP address of the device!

3D Printing an Enclosure #

The final part of my build was to create a small enclosure for the device that would house the antenna correctly. I found a nice model on Printables, which printed pretty fast on my BambuLab X1 Carbon printer.

The photo below shows the device in place on the window sill, with the hot tub just visible in the background:

Home Assistant Dashboard #

With all this in place, all that remained was tying the information into my Home Assistant dashboard:

The above shows the hot tub temperature, as well as the battery level of the IBS-P01 sensor. Beneath, I’ve added basic controls and measurements available through the Tapo Home Assistant integration, which allows me to quickly toggle the pump and heater, and see their daily energy usage at a glance.

If you look closely, you’ll see the room temperatures resulting from the Roth underfloor heating integration I wrote about previously.

Summary #

We’re super pleased with our tub - it was built by Andy from Rustic Tubs. If you’re considering one and you’re in the UK, you could do a lot worse. It’s beautifully crafted, and Andy was really helpful and communicative throughout the process.

This was my first foray into ESP32-based projects, and I was pleasantly surprised with how seamless the process was. The ESPHome docs were clear, and the no-hassle flashing through the browser was a nice way to get started.

I hope this post is useful to people who are just getting started, and feel free to reach out if I’ve missed something!

Writing a Home Assistant Core Integration: Part 2

Wed, 16 Oct 2024 00:00:00 +0000

Introduction #

In my last post I described the first steps toward creating a new Home Assistant integration for the underfloor heating system in my house. In that post I outlined in detail how I set about creating a Python client for the API provided by the underfloor heating controller vendor.

In this post, I’ll describe the development setup, project structure and contribution process for building and landing a Home Assistant Core integration. I don’t consider myself an expert here, but I’ve documented my journey here in the hope that my experience might be useful to potential future contributors.

The finished integration can be seen in the Home Assistant docs under the name touchline_sl, and the code can be found on Github.

Development Setup #

The Home Assistant documentation recommends the use of Visual Studio Code with a devcontainer. This was a very quick way to get a working environment up and running, especially given that I already use Visual Studio Code for most of my programming, so I was immediately familiar.

The repository provides some tasks to help get started, including a task named Run Home Assistant Core, which takes care of setting up the runtime environment, installing dependencies and starting the server. Neat!

There are also a set of pre-commit hooks set up to ensure you don’t make any common mistakes, accidentally violate the formatting/static-typing rules for the project, forget to update requirements.txt files, etc.

This turned out to be a really nice way to get started, and if you’re new to Home Assistant Core development, I’d recommend giving this “batteries-included” approach a go. If it’s not for you, the project provides manual setup instructions too.

Integration Basics #

According to the documentation:

[An] integration is responsible for a specific domain within Home Assistant. Integrations can listen for or trigger events, offer actions, and maintain states.

Where a domain is…

a short name consisting of characters and underscores. This domain has to be unique and cannot be changed.

Err, right… while this is an accurate statement, it’s perhaps not the most enlightening for the budding new integration developer! At their core, integrations are Python modules that take information about a given system (like an underfloor heating system, or a smart plug, or a light bulb) and represent information about them in a format compatible with one of Home Assistant’s archetypes for devices/sensors/platforms/entities.

In this case, we’ll be representing Climate Entities, which have the sort of properties you might expect - target_temperature, current_humidity, current_temperature, etc.

To use Home Assistant’s terms, in my setup:

The platform is the underfloor heating system
The platform consists of some devices, in this case the physical thermostats in each room of my house
The devices each represent one or more climate entities (humidity, temperature, etc.)

File Structure #

The basic file structure can be laid down with some scaffold tooling, but even in its finished state, my integration doesn’t have many files:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


.
├── __init__.py # the component file
├── climate.py # ties info from the api into home assistant terms
├── config_flow.py # defines the fields/flow for integration config
├── const.py # constants used across the integration
├── coordinator.py # data update coordinator
├── manifest.json # defines project dependencies and metadata
├── strings.json # defines strings displayed in various ui elements
└── translations # a directory containing one file per language
 └── en.json # english translation of strings.json

And to give an idea of the scale of the project in its completed form:

----------------------------------------------------------------------
Language files blank comment code
----------------------------------------------------------------------
Python 5 71 27 215
JSON 3 0 0 82
----------------------------------------------------------------------
SUM: 8 71 27 297
----------------------------------------------------------------------

`manifest.json` #

Starting with the most simple first! The manifest.json describes the integration: what its name is, where its documentation is found, who owns the code and the libraries it depends on:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


{
 "domain": "touchline_sl",
 "name": "Roth Touchline SL",
 "codeowners": ["@jnsgruk"],
 "config_flow": true,
 "documentation": "https://www.home-assistant.io/integrations/touchline_sl",
 "integration_type": "hub",
 "iot_class": "cloud_polling",
 "requirements": ["pytouchlinesl==0.1.8"]
}

Here I selected cloud_polling as the iot_class, because my integration reaches out periodically to the API, polling for new information. You’ll note also that the integration requires my pytouchlinesl library in order to run.

`init.py` #

Typically for integrations, this file defines how to setup the integration, and how to unload it. You can see the full source of my implementation on Github.

I define an async_setup_entry() method which takes care of:

Establishing a coordinator per TouchlineSL module
Performing the initial hydration of data
Registering each TouchlineSL module as a device in Home Assistant’s Device Registry, which is where Home Assistant “keeps track of devices”

There is a shortened, annotated version of the method below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


async def async_setup_entry(hass: HomeAssistant, entry: TouchlineSLConfigEntry) -> bool:
 """Set up Roth Touchline SL from a config entry."""
 account = TouchlineSL(...)

 coordinators: list[TouchlineSLModuleCoordinator] = [
 TouchlineSLModuleCoordinator(hass, module)
 for module in await account.modules()
 ]
 # ...
 device_registry = dr.async_get(hass)
 # Create a new Device for each coorodinator to represent each module
 for c in coordinators:
 module = c.data.module
 device_registry.async_get_or_create(
 config_entry_id=entry.entry_id,
 identifiers={(DOMAIN, module.id)},
 name=module.name,
 # ...
 )

 entry.runtime_data = coordinators
 await hass.config_entries.async_forward_entry_setups(entry, PLATFORMS)
 return True

Before returning async_setup_entry() invokes async_forward_entry_setups(). This ensures that async_setup_entry() in climate.py is called to ensure each of the climate entities is registered.

The PLATFORMS variable is a list of platform types that the integration supports, in this case a single-item list containing just Platform.CLIMATE, which is how async_forward_entry_setups knows to invoke the async_setup_entry() method in climate.py:

1
2
3
4


from homeassistant.const import Platform
# ...
PLATFORMS: list[Platform] = [Platform.CLIMATE]
# ...

To initialise the integration, the user must authenticate with the Roth API so that module details are fetched before constructing the coordinator. Thus, before this code is executed, the user must go through the config flow…

`config_flow.py` #

Despite the final implementation being quite simple, this is probably one of the areas I found most challenging to get right. There are some docs but they only scratch the surface, and implementations in other integrations seem to vary quite dramatically (mostly depending on when they were written).

I went through a few iterations of this config flow, mostly because I had originally implemented the ability to select a specific module from the user’s account. The review process guided me toward simply logging into the account, then enrolling each of the modules associated with the account - since users can always disable entities they don’t wish to manage in Home Assistant.

The config_flow.py defines which input fields need to be presented to the user, and then passes on the relevant information needed to set up the integration. In my implementation, the code:

Prompts the user for their username and password
Authenticates with the service and fetches the user’s unique ID
Registers that unique ID, aborting if the specified account has already been used
Creates a config entry in Home Assistant that stores the user’s credentials

The config flow also has some basic error handling that can distinguish the difference between poor credentials, networking issues, etc. The full implementation can be seen on Github, but the important parts are highlighted in the following snippet:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


# ...
class TouchlineSLConfigFlow(ConfigFlow, domain=DOMAIN):
 # ...
 async def async_step_user(
 self, user_input: dict[str, Any] | None = None
 ) -> ConfigFlowResult:
 # ...
 if user_input is not None:
 try:
 account = TouchlineSL(
 username=user_input[CONF_USERNAME],
 password=user_input[CONF_PASSWORD],
 )
 # Use the credentials to fetch unique user id
 await account.user_id()
 except RothAPIError as e:
 # ...
 else:
 # Set unique ID, abort setup if already used
 unique_account_id = await account.user_id()
 await self.async_set_unique_id(str(unique_account_id))
 self._abort_if_unique_id_configured()

 # Create a config entry containing the user's credentials
 return self.async_create_entry(
 title=user_input[CONF_USERNAME], data=user_input
 )

 return self.async_show_form(
 step_id="user", data_schema=STEP_USER_DATA_SCHEMA, errors=errors
 )

What caught me out was how the fields are given titles/descriptions. These attributes are all configured in the strings.json, where the config map contains keys for each of the config “steps”.

The code above defines a step named user, since the method name is async_step_user. The step’s name, description and input fields are defined in the strings.json:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


{
 "config": {
 "flow_title": "Touchline SL Setup Flow",
 "error": {
 "cannot_connect": "[%key:common::config_flow::error::cannot_connect%]"
 // ...
 },
 "step": {
 "user": {
 "title": "Login to Touchline SL",
 "description": "Your credentials for the Roth Touchline SL mobile app/web service",
 "data": {
 "username": "[%key:common::config_flow::data::username%]",
 "password": "[%key:common::config_flow::data::password%]"
 }
 }
 },
 "abort": {
 "already_configured": "[%key:common::config_flow::abort::already_configured_device%]"
 }
 }
 // ...
}

The values displayed to the user are pulled from the translation files at runtime depending on their language configuration. In my integration, the corresponding translations/en.json contains the following fields that map to those defined above:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


{
 "config": {
 "flow_title": "Touchline SL Setup Flow",
 "error": {
 "cannot_connect": "Failed to connect"
 // ...
 },
 "step": {
 "user": {
 "data": {
 "password": "Password",
 "username": "Username"
 },
 "description": "Your credentials for the Roth Touchline SL mobile app/web service",
 "title": "Login to Touchline SL"
 }
 },
 "abort": {
 "already_configured": "Device is already configured"
 }
 }
 // ...
}

In hindsight, this is covered in the docs, but it definitely didn’t click with me when I was going through it, so it seemed worth calling out! The net result of this setup is a configuration dialog that looks like so:

`coordinator.py` #

This was an addition I made during the review process (more on that later), and appears to be the preferred way to implement the fetching of data from upstream APIs. By implementing a DataUpdateCoordinator, Home Assistant can ensure that a single coordinated poll happens across all entities managed by an integration. If an integration manages many entities for which it needs to fetch/update details, the coordinator helps ensure that the API is called only as often as is needed.

The coordinator class is very simple: it defines a single method _async_update_data which returns the data for the device its coordinating. As the developer, you can specify the type of the data returned by the coordinator. I chose to represent this as a Python dataclass:

1
2
3
4
5


@dataclass
class TouchlineSLModuleData:
 module: Module
 zones: dict[int, Zone]
 schedules: dict[str, GlobalScheduleModel]

The coordinator is initialised with some basic information such as a name and an update interval:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# ...
class TouchlineSLModuleCoordinator(DataUpdateCoordinator[TouchlineSLModuleData]):
 def __init__(self, hass: HomeAssistant, module: Module) -> None:
 super().__init__(
 hass,
 logger=_LOGGER,
 name=f"Touchline SL ({module.name})",
 update_interval=timedelta(seconds=30),
 )

 self.module = module

The _async_update_data method then queries the API, and returns data in the newly defined format:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


async def _async_update_data(self) -> TouchlineSLModuleData:
 try:
 zones = await self.module.zones()
 schedules = await self.module.schedules()
 except RothAPIError as error:
 # ...
 # Return the data using our dataclass from above
 return TouchlineSLModuleData(
 module=self.module,
 zones={z.id: z for z in zones},
 schedules={s.name: s for s in schedules},
 )

You can see the full implementation on Github.

`climate.py` #

And finally, on to the business logic of tying fields from the upstream API into the relevant attributes in Home Assistant!

The first task handled by this file is registering each of the climate entities by iterating over each zone, in each coordinator’s module:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# ...
async def async_setup_entry(
 hass: HomeAssistant,
 entry: TouchlineSLConfigEntry,
 async_add_entities: AddEntitiesCallback,
) -> None:
 """Set up the Touchline devices."""
 coordinators = entry.runtime_data
 async_add_entities(
 TouchlineSLZone(coordinator=coordinator, zone_id=zone_id)
 for coordinator in coordinators
 for zone_id in coordinator.data.zones
 )
# ...

Home Assistant entities have well-defined APIs - the docs for climate entities show the supported attributes and their data types. I used a combination of the docs, and the source code to establish how to implement my ClimateEntity, which boiled down to the following interface:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


class TouchlineSLZone(CoordinatorEntity[TouchlineSLModuleCoordinator], ClimateEntity):
 # Construct a Touchline SL climate zone.
 def __init__(self, coordinator: TouchlineSLModuleCoordinator, zone_id: int) -> None:
 # Handle updated data from the coordinator.
 @callback
 def _handle_coordinator_update(self) -> None:
 # Return the device object from the coordinator data.
 @property
 def zone(self) -> Zone
 # Report if the device is available.
 @property
 def available(self) -> bool
 #Set new target temperature.
 async def async_set_temperature(self, **kwargs: Any) -> None
 # Assign the zone to a particular global schedule.
 async def async_set_preset_mode(self, preset_mode: str) -> None
 # Populate attributes with data from the coordinator.
 def set_attr(self) -> None:

Arguably the most important part here is set_attr(). which takes care of mapping fields from the objects provided by my pytouchlinesl library to attributes on the climate entity:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


def set_attr(self) -> None:
 """Populate attributes with data from the coordinator."""
 schedule_names = self.coordinator.data.schedules.keys()

 self._attr_current_temperature = self.zone.temperature
 self._attr_target_temperature = self.zone.target_temperature
 self._attr_current_humidity = int(self.zone.humidity)
 self._attr_preset_modes = [*schedule_names, CONSTANT_TEMPERATURE]

 if self.zone.mode == "constantTemp":
 self._attr_preset_mode = CONSTANT_TEMPERATURE
 elif self.zone.mode == "globalSchedule":
 schedule = self.zone.schedule
 self._attr_preset_mode = schedule.name

The full implementation is on Github.

Testing #

Testing is (understandably) a little complicated. The requirements for landing code for an integration stipulate that you must include unit tests for any config/options flows. In my case, this meant the following:

Implementing a mock TouchlineSL client fixture to ensure that the unit tests don’t try to reach out to the real Roth API
Implementing a mock config entry fixture
A unit test for a successful config flow execution
A parametrised unit test for unsuccessful config flow due to possible exceptions when hitting the API
A unit test to ensure that multiple config flows resulting in the same user ID fail

To me this feels like the bare minimum, and unfortunately doesn’t really provide any confidence that the integration actually functions correctly. I’m hoping to improve this in the future, but for now further testing has been manual.

Docs & Brand #

As I was creating the Pull Request to contribute my integration, I was prompted by the template to link to further PRs that added the documentation and brand assets for my integration. I used the existing touchline docs as a template and modified them for my integration. The docs pull request added a single file touchline_sl.markdown containing 29 lines, which results in some nicely rendered docs on the Home Assistant website:

There were already branding assets for Roth as part of the touchlinesl integration that was previously merged. Based on some review feedback, I created a new Integration Brand named roth, with which both the touchline and touchline_sl integrations are associated. This has the nice effect of grouping them when setting up a new integration:

Contribution Process #

I submitted my first efforts for review in home-assistant/core#124557. The checklist guided me nicely through what needed to be done, and overall the process went pretty smoothly, and pretty quickly (despite the 117 comments!). The process took a little under two days in total. I was also fortunate with my timing, since my code landed the day before the next beta release was cut, so it shipped relatively quickly.

I’d like to offer my sincere thanks to @joostlek who not only reviewed my code extremely quickly, but took the time to explain things to me both in the Github PR, but also by proactively reaching out to me on the Home Assistant Discord, which I really appreciated.

I got a lot of feedback, which I expected. This was my first attempt at writing code for Home Assistant, and it’s a pretty large and well established project. I do think the project could do with some better developer documentation, which would dramatically reduce the burden of effort on reviewers and give contributors a better chance of “getting it right”.

One might argue I could have done more reading and more research - though I spent a good amount of time reading both the documentation and the source code of other integrations. I’m not sure I’d ever have reached the conclusions that @joostlek kindly nudged me toward.

Overall, my implementation was more brief, more simple and more efficient as a result of the review process, and based on my experience I’d advocate for having a go if you’ve been on the fence! Often submitting code to such a project can be daunting, but as with my experience when contributing to nixpkgs, if you go in with an open mind you’re sure to learn something from the process.

Results #

I’m really pleased with the results. However obtuse the developer experience felt at times, there is no denying that what I was able to get from my 297 lines of implementation code is quite staggering. I was super impressed that just by following the conventions I was able to get such intuitive controls, a full graphable history of temperatures and such a wide variety of ways to display the information in my various dashboards.

Once set up, you’re able to get a view of all the different zones imported by the module (the names of each zone are pulled from the upstream API if the zones are named, and each can be associated with a given area in Home Assistant):

The default dashboard displays thermostat controls for each zone. These allow you to see the current and target temprature, as well as adjust the target temperature if you need to:

Expanding the thermostat cards gives you a more detailed view, showing the mode and the “Preset”. My implementation maps “Presets” to “Global Schedules” configured in the Roth module:

Summary #

And that’s a wrap! I learned a bunch writing this integration, and the resulting user experience is quite a lot better than I get with the default Roth application.

My next mission is to use the information from these climate entities to automate opening the Velux windows in the roof when things get warm in the summer, but I’ve got a lot to learn about Home Assistant in the mean time!

Writing a Home Assistant Core Integration: Part 1

Wed, 11 Sep 2024 00:00:00 +0000

Introduction #

Back in March, my family and I moved into a new home. It’s a modern construction which came with solar panels (and associated inverter/battery storage), and uses an air source heat pump to heat the house with underfloor heating. Being a new renovation, nearly all of the appliances and components in the house have a form of internet connectivity (some more useful than others!).

Since day 1, I’ve been hoping to consolidate all of the various applications, data feeds and functions into one single place. I’ve been a long-time listener to the Self Hosted podcast, which often extols the virtues of Home Assistant. I’ve got no prior experience with Home Assistant, but for the last three months I’ve been running it on my home server, with a collection of custom integrations and hacks that enable me to control the underfloor heating and solar inverter.

The underfloor heating controller is a Roth Touchline SL system. In my set up, there is a single “module” which represents my house, and a number of “zones” which represent different rooms.

There was unfortunately no integration for this system in Home Assistant - there is one for the previous generation “Roth Touchline”, but this appears to function over the LAN, whereas the Touchline SL system is controlled over the internet using their API.

After reading the source code of a few other climate integrations in Home Assistant, it became clear to me that the first step was to create a Python client for the API which could be used in the integration.

This post will cover the design, implementation and limitations of the library I wrote: pytouchlinesl. If you came here to read about writing code for Home Assistant, you’ll have to wait for the next post! 😉

Designing the library #

Upstream API #

Usually, one would interact with a Roth Touchline SL system through their mobile apps, or through their online portal. The mobile app seems to be quite a lightweight wrapper around the web application, and I’ve not been able to detect any difference in functionality.

A bit of searching uncovered that Roth also provide an API for the Touchline SL system, and an OpenAPI spec. This made the process significantly easier, though there are some discrepancies in what is documented compared with how the API actually behaves. It feels to me like the API may have evolved, and the documentation has remained static - or perhaps it was always inaccurate? Either way, I spent quite a bit of time manually fuzzing the API to work out the correct set of parameters for some endpoints.

I also studied the web application using the Chrome Dev Tools. Of all the endpoints documented, it seemed like I’d only need the following:

POST /authentication: authenticates with the API, taking a username and password, and returning a token
GET /users/{user_id}/modules: returns a list of modules associated with the user
GET /users/{user_id}/modules/{module_udid}: returns details of a specific module (zones, schedules, etc.)

There is a slight awkwardness here, in that the last endpoint returns all of the data - for all zones, all schedules, etc. This feels inefficient, but I couldn’t find a way of getting information about a specific zone, or a specific schedule. The web application seems to rely upon polling the (undocumented) endpoint GET /users/{user_id}/modules/{module_id}/update/data/parents/[]/alarm_ids/[]/last_update/{timestamp}, which delivers deltas in the data since a given timestamp. This is useful in the context of the app because it can request the full dataset once, then request only changes from that point onwards, keeping the app state up to date without requesting the whole dataset.

Making changes to the configuration of zones and their temperatures is also fragmented. In essence, one can:

Set a zone to a constant temperature: POST /users/{user_id}/modules/{module_udid}/zones
Place a zone on a global schedule: POST /users/{user_id}/modules/{module_udid}/zones/{zone_id}/global_schedule
Place a zone on a local schedule: POST /users/{user_id}/modules/{module_udid}/zones/{zone_id}/local_schedule

The first is self-explanatory, enabling a zone to be set to a constant temperature (19.0C, for example). Touchline SL modules also support “schedules” which contain time periods for the specified zones to reach certain temperatures. In the case of a “Global Schedule”, multiple zones can be assigned, while a “Local Schedule” is specific to a single zone. The awkwardness in the API here is that to “add” a zone to a global schedule, you must re-specify the entire schedule, and specify all of the zones that should be on the schedule…

Basic Requirements #

In order to fulfil the basic functionality of my (future) Home Assistant integration, I limited the requirements of the first version to:

Authenticate with the API using a username and password
List modules associated with the account
Get a specific module
Get a specific zone
Get a list of global schedules
Get a specific global schedule
Assign a constant temperature to a zone
Assign a zone to a specific global schedule

I don’t use local schedules in my system, so I’ve omitted them for now, though updating the library to support them would be trivial.

Outline design/experience #

With those requirements in mind, I came up with a rough sketch of how I’d like the library to behave:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


tsl = TouchlineSL(username="foo", password="bar")
module = await tsl.module(id="deadbeef")

lounge = module.zone_by_name("Lounge")
kitchen = module.zone(1234)

# Properties should be available such as:
# lounge.current_temperature
# kitchen.humidity

await kitchen.set_temperature(temperature=20.0)

living_spaces = await tsl.schedule_by_name(schedule_name="Living Spaces")
await lounge.set_schedule(living_spaces.id)

And from that came a reasonable outline of the API for the library:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


class TouchlineSL:
 # Construct a class that represents a Touchline SL account
 def __init__( self, *, username: str | None = None, password: str | None = None)
 # Get a list of modules associated with the account
 async def modules(self) -> list[Module]
 # Get a specific module, by ID
 async def module(self, *, module_id: str) -> Module | None

class Module:
 # Get a list of zones from the module, optionally including disabled zones
 async def zones(self, *, include_off: bool = False) -> list[Zone]
 # Get a specific zone, by ID
 async def zone(self, zone_id: int) -> Zone | None
 # Get a specific zone, by name
 async def zone_by_name(self, zone_name: str) -> Zone | None
 # Get a list of global schedules
 async def schedules(self) -> list[Schedule]:
 # Get a specific schedule, by ID
 async def schedule(self, schedule_id: int) -> Schedule | None
 # Get a specific schedule, by name
 async def schedule_by_name(self, schedule_name: int) -> Schedule | None

class Zone:
 # Get the schedule the zone is assigned to
 def schedule(self) -> Schedule | None
 # Set the zone to a constant temperature
 async def set_temperature(self, temperature: float)
 # Assign the zone to a specific schedule
 async def set_schedule(self, schedule_id: int)

Note that after reading the code from other climate integrations in Home Assistant, it became clear to me that they favour the use of async libraries, and thus my library was designed to use asyncio from the start.

Python tools/libraries #

There are a couple of things I’ve found tiresome about Python over the years, but things do seem to be looking up. I’ve always found the package management and distribution to be awkward, and I can’t be the only one if the number of projects looking to target that problem is anything to go by (e.g. poetry, rye, uv, pdm, etc.).

Part of this seems to come from fractures in the community itself - there (still!) appears to be disagreements surrounding PEPs such as PEP-621 which introduced pyproject.toml as a way of managing project metadata and dependencies, with the maintainers of some high-profile and widely adopted libraries refusing to adopt it.

That said, there are a couple of things I’ve been meaning to try in anger, and this project was a good opportunity to do so:

`uv` #

Developed by Astral, uv is the “new shiny” at the time of writing, and I can understand why. Pitched as “Cargo, but for Python”, it aims to solve a myriad of problems in the Python ecosystem. uv can handle the download/install of multiple Python versions, the creation of virtual environments, running Python tools in a one-off fashion (like pipx), locking dependencies deeply in a project (by hash) and still maintains a pip compatible command-line experience with uv pip. To add to all of that, it’s ridiculously fast; on a couple of occasions I’ve actually found myself wondering if it did anything when installing dependencies for large projects, because it’s so much faster than I’m used to.

`pydantic` #

Pydantic is a data validation library for Python. It’s entirely driven by Python’s type-hints which means that you get nice integration with language servers. Pydantic allows you define data models in native Python, but emit standard JSON Schema docs for models. It’s integrated quite widely across the Python ecosystem, and to me feels like it bridges the gap between what I hoped type annotations would do for Python, and what they actually do in reality!

`ruff` #

I’ve been using this one for at least the last year for all my Python linting and formatting needs, but I still feel it deserves a call out. There have been a couple of small changes to command line API and such along the way, but overall I’ve found it to be a dramatic improvement over my last setup - which comprised of black, isort, flake8 and a pile of plugins. I had no particular beef with black, but I find flake8’s lack of pyproject.toml support irritating, and grew tired of plugins failing as flake8 released new versions.

In my experience, ruff is stupid fast, and because it ships flake8-compatible rules for all of the plugins I was using as one bundle, they never break. It’s also nice to just have one tool to use everywhere. If you’re interested, you can see how I configure ruff in the pyproject.toml.

Implementation #

With a basic design in mind, and tooling ready to go, I set about building the library itself. It was now time to reconcile my intended design with the realities of the provisions made by the upstream API.

I mentioned previously that the only useful endpoint for getting information about zones/schedules would in fact return all of the data for a given module. Too many calls to this endpoint would likely result in poor performance, so I wanted to introduce some basic caching along the way.

Client implementation #

For the underlying API client implementation, I opted for the following:

BaseClient: a class which inherits from Python’s abc.ABC. This enables the creation of multiple client implementations by defining of the set of methods/properties that any client interacting with the Roth API should define. This decision is primarily to support testing through dependency injection, rather than mocking with patches (more details on that later).
RothAPI: a concrete implementation of the BaseClient abstract class. It is here that I built the actual implementation of the client which handles authentication, GETing and POSTing data, caching, and marshalling API responses into the correct types (defined with Pydantic).

Also included in the client package is the models package. The models package contains (mostly) auto-generated Pydantic models based on real-life responses I got from the API. This is really a function of laziness, but was a convenient way to get type annotated models for the responses I was receiving from the API. Each time I hit a new endpoint, I took the JSON result and did a quick conversion with https://jsontopydantic.com/, before manually adjusting names and updating some fields with Literals.

Caching #

I mentioned earlier that I wanted to implement some basic caching. While I am aware of various plugins for aiohttp (and other request libraries) that could handle this for me, my requirements were quite simple, so I chose to just build it into the library. In this case, caching is implemented on the Module class. This is because the large blob of data that is requested to populate details about a module and its zones/schedules is requested per module.

The caching works like this:

The Module class has “private” attributes named _raw_data and _last_fetched:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


class Module:
 def __init__(
 self,
 *,
 client: BaseClient,
 module_data: AccountModuleModel,
 cache_validity: int = 30,
 ):
 # ...

 # Raw data about the zones, schedules, tiles in the module
 self._raw_data: ModuleModel
 # Unix timestamp representing the last time the _raw_data was fetched
 self._last_fetched = 0
 self._cache_validity = cache_validity

 # ...

There is only one method on this class that calls the underlying client, and that’s another “private” method named _data. This method takes an optional refresh keyword argument, which forces the _raw_data attribute to be updated, but by default will only fetch data if the cached data has expired (after the number of seconds specified in self._cache_validity). If refresh is false, and the cache hasn’t expired, it simply returns the stored raw data:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


async def _data(self, *, refresh: bool = False) -> ModuleModel:
 """Get the raw representation of the module from the upstream API.

 If the data has never been fetched from upstream, or the data is older
 than the cache validity period, then the data is refreshed using the
 upstream API.

 Args:
 refresh: (Optional): Force the data to be refreshed using the API.
 """
 if refresh or (round(time.time() * 1000) - self._last_fetched) > self._cache_validity:
 self._raw_data = await self._client.module(self.id)
 self._last_fetched = round(time.time() * 1000)

 return self._raw_data

Each of the public methods (zones(), zone(), schedule(), etc.) access the raw data through the _data() method, and pass through the refresh flag which is exposed. This means that any developer consuming this library can chose to live with the caching, or override it and force a refresh like so:

1
2
3
4
5
6
7


tsl = TouchlineSL(username="foo", password="bar")
module = await tsl.module(id="deadbeef")

# Request a zone, accepting cached data (default)
zone = await module.zone_by_name("kitchen")
# Or force the data to be refreshed using the upstream API
zone = await module.zone_by_name("kitchen", refresh=True)

Testing, CI & Publishing #

Testing #

In the previous section, I described how I’d set up an abstract base class for the API client, then created an implementation of that for the actual Roth API. One of the main reasons I like this pattern for API clients is that it simplifies testing and reduces the need for monkey patching or traditional mocking (yes, I know the fake API client is sort of a mock…).

Because TouchlineSL can optionally be constructed with any client that implements BaseClient’s abstract base class, it’s trivial to implement a fake API backend that returns fixture data to be used in testing. In this case, the fixtures are stored as JSON files in the repository, and contain real life responses received from the API.

The FakeRothAPI returns the sample data for each of the methods defined in the abstract class. The following is a partial extract from the fake client code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


# ...
from pytouchlinesl.client import BaseClient
from pytouchlinesl.client.models import AccountModuleModel, ModuleModel

data_dir = Path(os.path.realpath(__file__)).parent / "sample-data"


class FakeRothAPI(BaseClient):
 def __init__(self):
 self._user_id = 123456789
 self._token = "deadbeef"

 async def user_id(self) -> int:
 return self._user_id

 async def authenticated(self) -> bool:
 return True if self._token else False

 async def modules(self) -> list[AccountModuleModel]:
 with open(data_dir / "modules.json", "r") as f:
 data = json.load(f)

 return [AccountModuleModel(**m) for m in data]

#...

From there, I defined a number of test fixtures using pytest’s @pytest.fixture decorator, which provide an initialised TouchlineSL instance, backed by the fake client, a Module instance, and a Zone instance.

From there, the tests are fairly simple. Beyond injecting the fake client, there is no mocking required, which in my opinion keeps the test code much easier to read and understand. It also means I could focus more of my energy on validating the logic I was testing, rather than worrying about how patching might interact with the rest of the code. I’ve felt bad about mocking for years, and I think the clearest articulation I’ve seen is “Don’t Mock What You Don’t Own” in 5 Minutes.

One aspect of the test suite I don’t love is the use of time.sleep in certain tests. This is because my caching implementation relies on reading a timestamp to decide on whether it should refresh data. In general I steer away from sleeps in tests, as they’re often used to mask an underlying non-determinism, but in this case it felt a reasonable trade-off, given that I’m testing a time-based functionality.

CI #

I wanted to ensure that any pull requests were tested, and that they conform to the project’s formatting/linting rules. Since the project is hosted on Github, I used Github Actions for this. The pipeline for this project is pretty simple, it lints and formats the code with ruff failing if any files were changed by the formatter or any of the linting rules were violated.

Finally, uv is used to run pytest across a matrix of supported Python versions. I could have used uv to handle the download and install of different Python versions too, but the setup-python actions has served me perfectly well in the past.

Publishing #

Publishing is taken care of in CI. I don’t like having to remember the magic incantation for building, authenticating and publishing locally. I want the process to be as consistent and as transparent as possible for the people consuming the project.

What was new to me this time was publishing to PyPI with a “Trusted Publisher” setup. To quote their docs:

“Trusted publishing” is our term for using the OpenID Connect (OIDC) standard to exchange short-lived identity tokens between a trusted third-party service and PyPI. This method can be used in automated environments and eliminates the need to use manually generated API tokens to authenticate with PyPI when publishing.

Rather than setting up Github Actions to hold an API token in a Secret, there is some automation that links a Github project to a PyPI project. The project doesn’t even have to be previously published on PyPI to get started, and new projects can be configured as “Pending”, then published to for the first time from your CI system of choice 🚀.

I configured Github Actions to trigger the release of pytouchlinesl on new tags being pushed:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


# ...
publish:
 name: Build and Publish to PyPI
 runs-on: ubuntu-latest
 needs:
 - tests

 # Trusted publisher setup for PyPI
 environment:
 name: pypi
 url: https://pypi.org/p/pytouchlinesl
 permissions:
 id-token: write

 steps:
 - name: Checkout the code
 uses: actions/checkout@v4

 - name: Install `uv`
 run: |
 curl -LsSf https://astral.sh/uv/install.sh | sh

 - name: Build the package
 run: |
 uvx --from build pyproject-build --installer uv

 - name: Publish to PyPi
 uses: pypa/gh-action-pypi-publish@v1.10.0

Contributing to `nixpkgs` #

Finally, my Home Assistant server runs NixOS, so in order for the integration to be packaged easily in the future, I created a small PR to include pytouchlinesl in nixpkgs. The original PR went through pretty quickly - thanks to @drupol for the fast review!

Since then I’ve made a couple of minor version bumps to the library as I discovered small issues when building the integration, but the final derivation is quite compact (see below). The Python build tooling in Nix is quite mature at this point, and I gained a fair bit of experience using it when packaging snapcraft and charmcraft.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32


#...

buildPythonPackage rec {
 pname = "pytouchlinesl";
 version = "0.1.5";
 pyproject = true;

 disabled = pythonOlder "3.10";

 src = fetchFromGitHub {
 owner = "jnsgruk";
 repo = "pytouchlinesl";
 rev = "refs/tags/${version}";
 hash = "sha256-kdLMuxA1Ig85mH7s9rlmVjEsItXxRlDA1JTFasnJogg=";
 };

 build-system = [ setuptools ];

 dependencies = [
 aiohttp
 pydantic
 ];

 nativeCheckInputs = [
 pytestCheckHook
 pytest-asyncio
 ];

 pythonImportsCheck = [ "pytouchlinesl" ];

 # ...
}

Summary #

And that concludes the first part of this series! Hopefully you found this useful - I’ve learned a lot from people over the years by understanding how they approach problems, so I thought I’d post my methodology here in case it helps anyone refine their process.

I’m certainly no Python expert, so if you’ve spotted a mistake or you think I’m wrong, get in touch.

In the next post, I’ll cover writing the Home Assistant integration, and contributing it to Home Assistant.

Libations: Tailscale on the Rocks

Wed, 21 Aug 2024 00:00:00 +0000

Introduction #

I’m a long-time, self-professed connoisseur of cocktails. I’ve always enjoyed making (and drinking!) the classics, but I also like experimenting with new base spirits, techniques, bitters etc.

Over the years, I’ve collected recipes from a variety of sources. Some originated from books (such as Cocktail Codex and Cocktails Made Easy), others from websites (Difford’s Guide), and most importantly those that I’ve either guessed from things I’ve drunk elsewhere (like my favourite bar in Bristol, The Milk Thistle), or modifications to recipes from the referenced sources.

I wanted somewhere to store all these variations: somewhere that I could search easily from my iPhone (which is nearly always close to me when I’m making drinks). I wanted each recipe to fit in its entirety on my iPhone screen without the need to scroll.

Around the time I started thinking about this problem, I also learned about tsnet and was desperate for an excuse to try it out - and thus Libations was born as the product of two things I love: Tailscale and cocktails!

tswhat? #

Some time ago, Tailscale released a Go library named tsnet. To quote the website:

tsnet is a library that lets you embed Tailscale inside of a Go program

In this case, the embedded Tailscale works slightly different to how tailscaled works (by default, anyway…). Rather than using the universal TUN/TAP driver in the Linux kernel, tsnet instead uses a userspace TCP/IP networking stack, which enables the process embedding it to make direct connections to other devices on your tailnet as if it were “just another machine”. This makes it easy to embed, and drops the requirement for the process to be privileged enough to access /dev/tun.

One of the things I like about how tsnet presents applications as devices on the tailnet, is that you can employ ACLs to control who and what on your tailnet can access the application, rather than the device. I’ve solved this problem before by putting applications in their own systemd-nspawn container and joining those containers to my tailnet. Another nice option is tsnsrv which essentially acts as a Tailscale-aware proxy for individual applications, but in this case I wanted to bake it into the application - which I would only access over my tailnet.

Getting started with tsnet couldn’t be easier:

1
2
3


mkdir tsnet-app; cd tsnet-app
go mod init tsnet-app
go get tailscale.com/tsnet

That will get you set up with a basic Go project, with the tsnet library available. Create a new main.go file with the following contents:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


package main

import (
 "fmt"
 "log"
 "net/http"

 "tailscale.com/tsnet"
)

func main() {
 // Create a new tsnet server instance
 s := tsnet.Server{Hostname: "tsnet-test"}
 defer s.Close()

 // Have the tsnet server listen on :8080
 ln, err := s.Listen("tcp", ":8080")
 if err != nil {
 log.Fatal(err)
 }
 defer ln.Close()

 // Define a very simple handler with a simple Hello, World style message
 handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 fmt.Fprintf(w, "<html><body><h1>Hello from %s, tailnet!</h1>\n", s.Hostname)
 })

 // Start an HTTP server on the tsnet listener
 err = http.Serve(ln, handler)
 if err != nil {
 log.Fatal(err)
 }
}

This is about the most minimal example I could contrive. The code creates a simple instance of tsnet.Server with the hostname tsnet-app, listens on port 8080 and serves up a simple Hello, World! style message. On running the application you’ll see the following:

1
2
3
4
5


❯ go run .
2024/08/13 15:03:37 tsnet running state path /home/jon/.config/tsnet-tsnet-app/tailscaled.state
2024/08/13 15:03:37 tsnet starting with hostname "tsnet-test", varRoot "/home/jon/.config/tsnet-tsnet-app"
2024/08/13 15:03:38 LocalBackend state is NeedsLogin; running StartLoginInteractive...
2024/08/13 15:03:43 To start this tsnet server, restart with TS_AUTHKEY set, or go to: https://login.tailscale.com/a/deadbeeffeebdaed

Clicking the link will open a page in your browser that runs you through Tailscale’s authentication flow, after which you should be able to curl the page directly from any of your devices (assuming you’re not doing anything complicated with ACLs that might prevent it)!

1
2
3
4
5
6
7
8


❯ tailscale status
# ...
100.93.165.28 kara jnsgruk@ linux -
100.106.82.10 tsnet-test jnsgruk@ linux -
# ...

❯ curl http://tsnet-test:8080
<html><body><h1>Hello from tsnet-test, tailnet!</h1>

The library has a pretty small API surface, all of which is documented on pkg.go.dev.

Libations #

For my cocktail app, I wanted to employ a similar, albeit simplified, set of techniques that I use to build this blog. I love Go’s ability to embed static files that can be served as web assets.

Recipe Schema #

I wanted to represent recipes in a format that could be updated by hand if necessary, and easily parsed into a web frontend. I decided to use JSON for this, representing the recipes as a list:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


[
 {
 "id": 10,
 "name": "New York Sour",
 "base": ["Bourbon"],
 "glass": ["12oz Lowball"],
 "method": ["Dry Shake", "Shake"],
 "ice": ["Cubed"],
 "ingredients": [
 { "name": "Lemon Juice", "measure": "20", "unit": "ml" },
 { "name": "Sugar", "measure": "20", "unit": "ml" },
 { "name": "Red Wine", "measure": "10", "unit": "ml" },
 { "name": "Bourbon", "measure": "40", "unit": "ml" },
 { "name": "Egg White", "measure": "20", "unit": "ml" }
 ],
 "garnish": ["Lemon Sail"],
 "notes": "Use claret or malbec"
 }
]

This schema is able to capture all the relevant details from the different formats I’ve seen over the years. It would take some time to format my favourites into this schema, but that was always going to be the case.

I was fortunate enough to get access to the recipe collection from a well regarded cocktail bar in the UK. Unfortunately it was given to me in a hard-to-parse PDF, which resulted in many hours of playing with OCR tools and manual data cleaning - but enabled me to bootstrap the app with around 450 high quality recipes. I didn’t include their recipes in the libations repository, but I did include some of my own favourite concoctions in a sample recipe file. My Mezcal Margarita gets pretty good reviews 😉.

Server #

The server implementation needed to fulfil a few requirements:

Parse a specific recipes file, optionally passed via the command line
Have an embedded filesystem to contain static assets and templates
Be able to render HTML templates with the given recipes
Listen on either a tailnet (via tsnet), or locally (for testing convenience)
When listening on the tailnet, listen on HTTPS, redirecting HTTP traffic accordingly

I wanted to keep dependencies to a minimum to make things easier to maintain over time. The tsnet library pulls in a few indirect dependencies, but everything else Libations uses is in the Go standard library.

The recipes JSON schema is very simple, and is modelled with a couple of Go structs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


// Ingredient represents the name and quantity of a given ingredient in a recipe.
type Ingredient struct {
 Name string
 Measure string
 Unit string
}

// Drink represents all of the details for a given drink.
type Drink struct {
 ID int
 Name string
 Base []string
 Glass []string
 Method []string
 Ice []string
 Ingredients []Ingredient
 Garnish []string
 Notes string
}

The parseRecipes function checks whether or not the user passed the path to a specific recipe file, or whether it should default to parsing the sample recipes file. Once it’s determined the right file to parse, and validated its existence, it unmarshals the JSON using the Go standard library.

Users have the option of passing the -local flag when starting Libations, which bypasses tsnet completely and starts a local HTTP listener on the specific port. This makes for easier testing when iterating through changes to the web UI and other elements:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


// ...
addr := flag.String("addr", ":8080", "the address to listen on in the case of a local listener")
local := flag.Bool("local", false, "start on local addr; don't attach to a tailnet")

// ...
var listener *net.Listener
if *local {
 listener, err = localListener(*addr)
} else {
 listener, err = tailscaleListener(*hostname, *tsnetLogs)
}
if err != nil {
 slog.Error("failed to create listener", "error", err.Error())
 os.Exit(1)
}
//...

Setting up the tsnet server and listener is only mildly more complicated – but mostly due to my requirement that all HTTP traffic is redirected to HTTPS, using the LetsEncrypt certificates that Tailscale provides automatically. The redirects are handled by a separate Goroutine in this case:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


// Start a standard HTTP server in the background to redirect HTTP -> HTTPS.
go func() {
 httpLn, err := tsnetServer.Listen("tcp", ":80")
 if err != nil {
 slog.Error("unable to start HTTP listener, redirects from http->https will not work")
 return
 }

 slog.Info(fmt.Sprintf("started HTTP listener with tsnet at %s:80", hostname))

 err = http.Serve(httpLn, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 newURL := fmt.Sprintf("https://%s%s", r.Host, r.RequestURI)
 http.Redirect(w, r, newURL, http.StatusMovedPermanently)
 }))
 if err != nil {
 slog.Error("unable to start http server, redirects from http->https will not work")
 }
}()

With the correct listener selected, I create an http.ServeMux to handle routing to static assets and rendering templates, and pass that mux to the http.Serve method from the Go standard library - and that’s it! At the time of writing the Go code totals 235 lines - not bad!

Web Interface #

The web interface was designed primarily for mobile devices, and I’ve not yet done the work to make it excellent for larger-screened devices - though it’s certainly bearable. It’s also read-only at the moment: you can browse all the recipes, and there is a simple full-text search which can narrow the list of recipes down by searching for an ingredient, method, glass type, notes, etc.

As mentioned in an earlier post, I’m a big fan of the Vanilla Framework, which is a “simple, extensible CSS framework” from Canonical, and is used for all of Canonical’s apps and websites. Given that I had some prior experience using it, I decided to use it again here. I started using my tried and tested recipe of Vanilla + Hugo, but later reverted to using simple HTML templates with Go’s html/template package.

The result is a set of templates, which iterate over the recipe data from the JSON file, and output nicely styled HTML elements:

There is nothing fancy going on here - it’s nearly all stock Vanilla Framework. I do specify some overrides to make the colours a bit less Ubuntu-ish, but that’s it!

One detail I’m pleased with is the dynamic drink icons. The icons indicate the type of glass the particular drink should be served in. This is a simple trick: for each drink the HTML template renders a glass-icon partial, which reads the glass type specified in the recipe, and renders the appropriate SVG file which is then coloured with CSS.

Packaging for NixOS #

There were two main tasks in this category: creating the Nix package itself, and writing a simple NixOS module that would make it simple for me to run it on my NixOS server.

The project uses a Flake to provide the package, overlay and module. The standard library in Nix has good tooling for Go applications now, meaning the derivation is short:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


{
 buildGo122Module,
 lastModifiedDate,
 lib,
 ...
}:

let
 version = builtins.substring 0 8 lastModifiedDate;
in
buildGo122Module {
 pname = "libations";
 inherit version;
 src = lib.cleanSource ../.;
 vendorHash = "sha256-AWvaHyJL7Cm+zCY/vTuTAsgLbVy6WUNfmaGbyQOzMMQ=";
}

I haven’t cut any versioned releases of Libations at the time of writing - I’m using the last modified date of the flake to version the binaries.

The module starts the application with systemd, and optionally provides it with a recipes file. There are four options defined at the time of writing: services.libations.{enable,recipesFile,tailscaleKeyFile,package}:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


options = {
 services.libations = {
 enable = mkEnableOption "Enables the libations service";

 recipesFile = mkOption {
 type = nullOr path;
 default = null;
 example = "/var/lib/libations/recipes.json";
 description = ''
 A file containing drinks recipes per the Libations file format.
 See https://github.com/jnsgruk/libations.
 '';
 };

 tailscaleKeyFile = mkOption {
 type = nullOr path;
 default = null;
 example = "/run/agenix/libations-tsauthkey";
 description = ''
 A file containing a key for Libations to join a Tailscale network.
 See https://tailscale.com/kb/1085/auth-keys/.
 '';
 };

 package = mkPackageOption pkgs "libations" { };
 };
};

The tailscaleKeyFile option enables the service to automatically join a tailnet using an API key, rather than prompting the user to click a link and authorise manually.

These options are translated into a simple systemd unit:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


config = mkIf cfg.enable {
 systemd.services.libations = {
 description = "Libations cocktail recipe viewer";
 wantedBy = [ "multi-user.target" ];
 after = [ "network.target" ];
 environment = {
 "XDG_CONFIG_HOME" = "/var/lib/libations/";
 };
 serviceConfig = {
 DynamicUser = true;
 ExecStart = "${cfg.package}/bin/libations -recipes-file ${cfg.recipesFile}";
 Restart = "always";
 EnvironmentFile = cfg.tailscaleKeyFile;
 StateDirectory = "libations";
 StateDirectoryMode = "0750";
 };
 };
}

The XDG_CONFIG_HOME variable is set so that tsnet stores it’s state in /var/lib/libations, rather than trying to store it in the home directory of the dynamically created user for the systemd unit.

I use agenix on my NixOS machines to manage encrypted secrets, and for this project I used it to encrypt both the initial Tailscale auth key, and my super-secret recipe collection! The configuration to provide the secrets and configure the server to run libations is available on Github, but looks like so:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


{ config, self, ... }:
{
 age.secrets = {
 libations-auth-key = {
 file = "${self}/secrets/thor-libations-tskey.age";
 owner = "root";
 group = "root";
 mode = "400";
 };

 libations-recipes = {
 file = "${self}/secrets/thor-libations-recipes.age";
 owner = "root";
 group = "root";
 mode = "444";
 };
 };

 services.libations = {
 enable = true;
 recipesFile = config.age.secrets.libations-recipes.path;
 tailscaleKeyFile = config.age.secrets.libations-auth-key.path;
 };
}

As a result, the application is now available at https://libations, with a valid LetsEncrypt certificate, on all of my machines! 🎉

Summary #

This was a really fun project. It felt like a fun way to explore tsnet, and resulted in something that I’ve used a lot over the past year. I don’t have many plans to adjust things in the near future - though I do find myself wanting a nice interface to add new recipes from time to time.

And now for the twist: three weeks ago I gave up drinking alcohol (likely for good), so now I’m on a mission to find some non-alcoholic recipes to make life a little tastier! I suspect this will be hard work - and I’ll certainly miss some of my favourites, but my wife and I have already found some compelling alternatives.

If you’ve got a favourite recipe (alcoholic or not) and you liked the article, then perhaps open a PR and add it to the sample recipes file!

A homelab dashboard for NixOS

Tue, 05 Mar 2024 00:00:00 +0000

Introduction #

I run a very small homelab that provides some basic services to my home network. I’m not much of a data hoarder, but my lab consists of some redundant storage in a raidz2 ZFS pool, and I use the homelab as a receive-only target for Syncthing, and as the point from which backups of my critical data are made using Borg & Borgbase.

It also runs a few other small services - all of which are exclusively available over Tailscale to my other devices. I wanted a small dashboard solution that could give me links to each of those services with a nice simple URL.

There are certainly plenty of options; this seems to be a highly crowded space in the open source homelab world. I settled on the rather ambiguously named homepage. At the time of writing, my dashboard looks like so, though there are people who have been far more creative with the appearance!

Naturally, I wanted to run this on NixOS, so in July 2023 I landed one of my early contributions to the project in the form of PR #243094 which added the package (named homepage-dashboard), a basic NixOS module and a basic test.

Homepage’s Configuration #

Homepage is configured using a set of YAML files named services.yaml, bookmarks.yaml, widgets.yaml, etc. When I originally started writing the module, it would look for those config files in a hard-coded location (/config), and if the files were missing it would copy a skeleton config into place with some defaults to get you going.

The hard-coded location results from the fact that the upstream primarily support deploying Homepage using Docker. They expect the config directory to be bind-mounted into the container from the host. As part of the initial packaging effort, I contributed a patch upstream to allow customising this location by setting the HOMEPAGE_CONFIG_DIR environment variable, which I then set in the systemd unit configuration in the NixOS module to /var/lib/homepage-dashboard.

This has been working fine for a few months, but it’s been bugging me that my dashboard configuration is not part of the declarative system configuration. Once the initial skeleton had been copied in place, you were left to edit the files manually (and back them up) if you wanted to make changes. Moreover, homepage defaults to creating a logs directory as a subdirectory of the config directory. This makes some sense in a container environment, but given that homepage also logs to stdout (which is collected by the systemd journal on NixOS), it’s really just unnecessary duplication.

Evolving The Module Design #

Before writing the actual implementation of the module, I decided to first sketch out what I wanted my NixOS configuration to look like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


{
 services.homepage-dashboard = {
 # These options were already present in my configuration.
 enable = true;
 package = unstable.homepage-dashboard;

 # The following options were what I planned to add.

 # https://gethomepage.dev/latest/configs/settings/
 settings = {};

 # https://gethomepage.dev/latest/configs/bookmarks/
 bookmarks = [];

 # https://gethomepage.dev/latest/configs/services/
 services = [];

 # https://gethomepage.dev/latest/configs/service-widgets/
 widgets = [];

 # https://gethomepage.dev/latest/configs/kubernetes/
 kubernetes = { };

 # https://gethomepage.dev/latest/configs/docker/
 docker = { };

 # https://gethomepage.dev/latest/configs/custom-css-js/
 customJS = "";
 customCSS = "";
 };
}

Each of the new sections would then map neatly to the different configuration section in the upstream documentation. Based on my learning from the Scrutiny module, I wanted to utilise the same RFC42 approach which would obviate the need for the module to specify every possible supported configuration option, resulting in a large, difficult to maintain module which could quickly fall behind the upstream project.

Homepage supports a large number of widgets (see below) which are able to scrape information from the API of various devices and services. These often require an API key or token of some kind, and having those in plaintext as part of the machine configuration is undesirable from a security perspective - even if your services are all on a private network like mine. Luckily I found out (tucked away in the docs) that Homepage can inject secret values into the configuration using environment variables.

Given the template value of {{HOMEPAGE_VAR_FOOBAR}} as part of the configuration, Homepage will automatically substitute the value of the variable HOMEPAGE_VAR_FOOBAR.

I decided to provide a single configuration option named environmentFile so that users can supply the path to an environment file containing all of their variables. This file can be omitted from Git repositories and configurations, or included in encrypted form. I achieve this by including the file encrypted using agenix which integrates @FiloSottile’s wonderful age into NixOS. You can see how that’s supplied as part of my nixos-config.

Backwards Compatibility #

According to my relatively naive Github search I estimated that there are not that many users of the module - likely in the tens, rather than the hundreds or thousands. That said, I think its important not to break those users. There’s no reason to expect that a nix flake update should break your system.

The way I chose to handle this in the module was to check if any of the new config options are set. If they’re not, the module behaves as before, but displays a deprecation warning:

The implementation of this check is relatively crude, but it works, and it will only be around until the release of NixOS 24.05 (in May 24):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


config =
 let
 # If homepage-dashboard is enabled, but none of the configuration values have been updated,
 # then default to "unmanaged" configuration which is manually updated in
 # var/lib/homepage-dashboard. This is to maintain backwards compatibility, and should be
 # deprecated in a future release.
 managedConfig = !(
 cfg.bookmarks == [ ] &&
 cfg.customCSS == "" &&
 cfg.customJS == "" &&
 cfg.docker == { } &&
 cfg.kubernetes == { } &&
 cfg.services == [ ] &&
 cfg.settings == { } &&
 cfg.widgets == [ ]
 );

 configDir = if managedConfig then "/etc/homepage-dashboard" else "/var/lib/homepage-dashboard";

 msg = "using unmanaged configuration for homepage-dashboard is deprecated and will be removed"
 + " in 24.05. please see the NixOS documentation for `services.homepage-dashboard' and add"
 + " your bookmarks, services, widgets, and other configuration using the options provided.";
 in
 lib.mkIf cfg.enable {
 # Display the deprecation warning if the configuration isn't managed
 warnings = lib.optional (!managedConfig) msg;

# ...

Solving Log Duplication #

I mentioned in a previous section that Homepage logs to both stdout and a logs directory by default. While the log file path can be customised, it’s not currently possible to disable the file logging completely. It’s not desirable to have the log file in this context, because all of the logs are collected by systemd anyway.

Looking at the upstream implementation, the logger is instantiated and configured in a single logger.js file. Homepage has a policy that they won’t accept feature contributions (even if you do the implementation) unless the feature gets at least 10 upvotes. I filed a feature request, but it’s yet to get enough votes to be accepted.

In the mean time I wrote a short patch on a branch in my personal fork which makes Homepage adhere to an environment variable named LOG_TARGETS. The possible values are both, file or stdout with a default value of both to respect the existing behaviour and remain backward compatible. The patch is now applied in the Nix package as part of nixpkgs, and the module configures the systemd unit by setting the LOG_TARGETS variable to stdout in cases where the configuration is managed:

1
2
3
4
5
6
7
8
9


{
 # ...
 environment = {
 HOMEPAGE_CONFIG_DIR = configDir;
 PORT = toString cfg.listenPort;
 LOG_TARGETS = lib.mkIf managedConfig "stdout";
 };
 # ...
}

Update: My feature request got upvoted pretty quickly, and as a result my pull request was merged. This means that following the next release of Homepage, I’ll be able to drop the patch from the Nix package. Thanks to all those who upvoted it!

Bolstering The Test Suite #

When I originally implemented the tests for the module, they simply enabled the service and ensure that it responded on the specified port. I wanted to include some logic in the test that ensured the ability to detect when managed configuration should be used, and when the module should respect an existing implementation.

The NixOS test suite supports specifying multiple machines as part of a given test, so extending the previous implementation wasn’t particularly cumbersome. See below for the (annotated) implementation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49


import ./make-test-python.nix ({ lib, ... }: {
 name = "homepage-dashboard";
 meta.maintainers = with lib.maintainers; [ jnsgruk ];

 # Create a machine that uses the legacy module format,
 # where configuration is unmanaged by nix, and relies
 # upon YAML files.
 nodes.unmanaged_conf = { pkgs, ... }: {
 services.homepage-dashboard.enable = true;
 };

 # Create another machine that sets some simple
 # configuration using the new module system. This
 # doesn't need to be exhaustive, just enough to trigger
 # the condition that makes the module use managed config.
 nodes.managed_conf = { pkgs, ... }: {
 services.homepage-dashboard = {
 enable = true;
 settings.title = "custom";
 };
 };

 testScript = ''
 # Ensure the services are started on unmanaged machine,
 # and that the service responds to HTTP requests on the
 # expected port.
 unmanaged_conf.wait_for_unit("homepage-dashboard.service")
 unmanaged_conf.wait_for_open_port(8082)
 unmanaged_conf.succeed("curl --fail http://localhost:8082/")

 # Ensure that /etc/homepage-dashboard doesn't exist, and boilerplate
 # configs are copied into place in `/var/lib/homepage-dashboard`.
 # This validates the existing behaviour.
 unmanaged_conf.fail("test -d /etc/homepage-dashboard")
 unmanaged_conf.succeed("test -f /var/lib/private/homepage-dashboard/settings.yaml")

 # Ensure the services are started on managed machine,
 # and that the service responds to HTTP requests on the
 # expected port.
 managed_conf.wait_for_unit("homepage-dashboard.service")
 managed_conf.wait_for_open_port(8082)
 managed_conf.succeed("curl --fail http://localhost:8082/")

 # Ensure /etc/homepage-dashboard is created and unmanaged
 # conf location isn't present
 managed_conf.succeed("test -d /etc/homepage-dashboard")
 managed_conf.fail("test -f /var/lib/private/homepage-dashboard/settings.yaml")
 '';
})

This is by no means exhaustive, and I can certainly imagine increasing the coverage here at a later date, but it does at least give some confidence when working on the module that the two basic modes of operation are functioning correctly.

Migrating Existing Configurations #

If you have been using the module in its past form, you may be wondering what the easiest way to migrate to the new format is…

I made the shift using yaml2nix to convert my existing YAML configurations to Nix expressions, and then formatted the output using nixpkgs-fmt. For example, given the following settings.yaml (which came from my homelab before I moved over):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


---
# For configuration options and examples, please see:
# https://gethomepage.dev/en/configs/settings

title: sgrs dashboard
favicon: https://jnsgr.uk/favicon.ico
headerStyle: clean

layout:
 media:
 style: row
 columns: 3
 infra:
 style: row
 columns: 4
 machines:
 style: row
 columns: 4

You can do the following to get a Nix expression that can be assigned to services.homepage-dashboard.settings in your machine configuration, converting the YAML to a Nix expression:

 ~/temp
❯ nix run nixpkgs#yaml2nix settings.yaml
{ title = "sgrs dashboard"; favicon = "https://jnsgr.uk/favicon.ico"; headerStyle = "clean"; layout = { media = { style = "row"; columns = 3; }; infra = { style = "row"; columns = 4; }; machines = { style = "row"; columns = 4; }; }; }

With that output, you can insert a few line breaks and rely on nixpkgs-fmt to get everything lined up properly. You can see my complete dashboard configuration in Nix format as part of my nixos-config repository.

Summary #

The PR was merged earlier today, and will now need to trickle through the branches on its way to nixos-unstable. At the time of writing, it hasn’t quite made it there:

You can track for yourself on the nixpkgs tracker, but the time delay should give you a chance to migrate your configuration!

See Github for the full module implementation, package and tests.

Let me know if you’re using the module, or if you run into any issues! If you’re a fan of Homepage, then consider helping out with the project or sponsoring them on Github, and once again thank you to those who helped review and shape the module as part of this upgrade!

Contributing Scrutiny to nixpkgs

Mon, 19 Feb 2024 00:00:00 +0000

Introduction #

I’m quite overwhelmed by the number of people who read, and engaged with my last post on packaging Scrutiny for NixOS. I hadn’t intended on posting to this blog more than maybe once per month, but I decided to continue documenting the journey of landing Scrutiny upstream in nixpkgs.

One of the things that really struck me about NixOS early on was how simple the contribution process is. A single Github repository contains all the packages, all the modules, and all the tests.

I’ve found the community incredibly helpful and respectful in all of my contributions - often with 2-3 people quickly providing reviews to help shape the contribution. I’ve heard complaints that reviews can be quite pedantic, and while I agree to some extent, the reviews nearly always come with concrete suggestions which are a great way to learn. Notwithstanding the fact that you are contributing to an operating system package that could land on many thousands of machines - it’s important to be careful!

If you’ve been deliberating about whether or not to submit your package, module or otherwise, I’d encourage you to go for it. I’ve always found it rewarding and it’s taught me a lot about Nix and NixOS.

My first contribution was adding a package, module and test for multipass - as you can see I had plenty to learn, but people were generous with their time and it resulted in a contribution that I use daily to get my work done on NixOS.

For those losing sleep over the poorly disk I showed in my last post, you’ll be relieved to know that it’s now been replaced! Incidentally this was the first time I’ve replaced a disk in a ZFS array and I was super impressed with how easy the process was!

Improving My Work #

Over the weekend following my first post about Scrutiny, a couple of people identified some small issues and made pull requests to resolve them - thank you to those people! You can see their work here, here and here. Someone also pointed out to me that string-interpolating YAML wasn’t the most ideal solution to rendering the configuration file, and the I could have used builtins.toJSON, taking advantage of the fact that JSON is also valid YAML.

I also wanted to make sure that there was a way for people to use the module with an existing InfluxDB deployment. My original implementation assumed that it could just add services.influxdb2.enable = true to the system configuration and use the default credentials to connect. I added some options to the module for configuring a different InfluxDB instance if required. These were later modified slightly to use lib.types.nullOr lib.types.str following a helpful review comment.

Next up was refactoring the helper function I wrote for generating the configuration file. As mentioned above, Nix comes with a handy builtins.toJSON function which takes a native Nix expression (an attribute set in this case) and renders it to JSON. This obviates the need to string-interpolate YAML, which is error prone at the best of times. The renewed function looked like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


{
 mkScrutinyCfg = cfg: pkgs.writeTextFile {
 name = "scrutiny.yaml";
 text = builtins.toJSON {
 version = 1;
 web = {
 listen = {
 inherit (cfg) host port;
 basepath = if cfg.basepath != "" then "/" + cfg.basepath else "";
 };

 database = {
 location = "/var/lib/scrutiny/scrutiny.db";
 };

 src = {
 frontend.path = "${pkgs.scrutiny}/share/scrutiny";
 };

 influxdb = {
 inherit (cfg.influxdb) scheme host port token org bucket;
 tls.insecure_skip_verify = cfg.influxdb.tlsSkipVerify;
 };
 };
 log = {
 level = cfg.logLevel;
 };
 };
 };
}

I also noticed that I had hard-coded the path to the binaries in the rendered systemd units. You might recall that each Nix package contains meta block, which adds information about the package such as the license, maintainer, etc. For each of the packages, I added a mainProgram attribute which enables the use of something like lib.getExe pkgs.scrutiny rather than ${pkgs.scrutiny}/bin/scrutiny.

1
2
3
4
5
6
7


{
 # ...
 meta = {
 mainProgram = "scrutiny";
 # ...
 };
}

Finally, I wanted to simplify the packaging of the Go programs slightly. In the previous post I used buildPhase to override the default build behaviour of the buildGoModule function. I later discovered the subPackages attribute, which enabled me to achieve the same effect without overriding the process manually. You can see the commit with all the changes on Github, but below is a (slightly modified) preview of the simplified collector derivation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


{ pkgs, lib, ... }:
pkgs.buildGoModule rec {
 # (source, name, version, vendor hash omitted)

 subPackages = "collector/cmd/collector-metrics/collector-metrics.go";

 buildInputs = with pkgs; [ makeWrapper ];

 CGO_ENABLED = 0;
 ldflags = [ "-extldflags=-static" ];
 tags = [ "netgo" "static" ];

 installPhase = ''
 mkdir -p $out/bin
 cp $GOPATH/bin/collector-metrics $out/bin/scrutiny-collector-metrics
 wrapProgram $out/bin/scrutiny-collector-metrics \
 --prefix PATH : ${lib.makeBinPath [ pkgs.smartmontools ]}
 '';

 # (meta block omitted)
}

Contributing To nixpkgs #

With these changes made, it was time to open a pull request. The NixOS community provide some contributing guidelines which are worth a read before you start.

There was a structural transition in the pkgs directory lately, so this was my first contribution under that new structure. There was a detailed talk about that transition at Nixcon if you’d like more information.

I needed to do the following:

Create the scrutiny and scrutiny-collector packages
Add the scrutiny module
Add the tests
Update the release notes for the next release of NixOS

Which translated into the following stack of commits on PR #289934:

There were a few other minor changes, such as how input packages (like smartmontools and makeWrapper) are passed to the packages. In the nixpkgs repository, packages are passed directly to the derivations, rather than pkgs being passed as a top-level argument and packages being referred to by pkgs.smartmontools etc.

I also learned something new about module configuration. In the first iteration I had set the default value of services.scrutiny.collector.enable to the value of services.scrutiny.enable, meaning that the collector would be enabled automatically when Scrutiny was enabled if no other configuration was specified. It transpires that this is not permitted in nixpkgs, and resulted in this CI failure. I updated the module with a subtle API change, meaning that consumers of the module will now need to do the following to get the same behaviour.

1
2
3
4
5
6


{
 services = {
 scrutiny.enable = true;
 scrutiny.collector.enable = true;
 };
}

Review Feedback #

From the first review I learned about the lib.types.nullOr meta-type which enables the specification of optionally null configuration options in a NixOS module.

What followed was a suggestion to embrace a different configuration format according to RFC42: a proposal for structured module configuration which is more rigorously type checked, and more flexible as new options are added to the underlying workload.

Many NixOS modules solve this problem today with options like extraConfig or configLines, which append strings to the end of any options-generated configuration file. What RFC42 proposes is a hybrid approach whereby certain options are well-defined (and thus feature in the documentation), but additional options can be injected as needed. The has the nice side-effect that modules do not need to support dozens of options which can become a burden to maintain and test over time, but consumers can still make use of new features and options that are introduced to the underlying workload.

You can see the full resulting options definition on Github, and small annotated excerpt below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


{
 services.scrutiny = {
 enable = lib.mkEnableOption "Enables the scrutiny web application.";

 # ...

 settings = lib.mkOption {
 description = lib.mdDoc ''
 Scrutiny settings to be rendered into the configuration file.

 See https://github.com/AnalogJ/scrutiny/blob/master/example.scrutiny.yaml.
 '';

 default = { };

 # This is the key to RFC42 - defining `settings` as type `submodule`.
 type = lib.types.submodule {
 # This ensures that attrsets, lists, etc. are rendered as JSON
 freeformType = (pkgs.formats.yaml { }).type;

 # The following are examples of "well-defined" options with
 # formal types and metadata.
 options.web.listen.port = lib.mkOption {
 type = lib.types.port;
 default = 8080;
 description = lib.mdDoc "Port for web application to listen on.";
 };

 options.web.listen.host = lib.mkOption {
 type = lib.types.str;
 default = "0.0.0.0";
 description = lib.mdDoc "Interface address for web application to bind to.";
 };

 # ...
 };
 };
}

This enables the following user configurations:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


{
 services.scrutiny = {
 enable = true;
 settings = {
 # These options are well-defined in the module with types,
 # examples, descriptions and are rendered in the docs.
 web.listen = {
 port = 8080;
 host = "100.64.12.34";
 };

 # These are not, but will be rendered still as JSON and enable
 # people to make use of options supported by Scrutiny, but
 # not directly by the module.
 notify = {
 urls = [
 "discord://token@channel"
 ];
 };
 };
 };
}

I was impressed by the simplicity of the approach, and subsequently adopted the same for the collector, which now has the ability to take configuration through the services.scrutiny.collector.settings attribute.

One aspect of Scrutiny’s design that helped is the ability to take configuration either through from file, or through environment variables, or both! I chose to supply some fixed options as environment variables to the systemd services, and allow the rest of the configuration to be set using the generated file.

@JulienMalka reviewed next, suggesting changes to ensure that various preInstall and postInstall hooks were being called where I was overriding installPhase, though what followed from @katexochen’s review was a simplification that reduced this need slightly.

@katexochen suggested a couple of nice changes, among which was the ability to drop explicitly mentioning the netgo tag in the Go packages, since CGO_ENABLED is disabled already, this implies using the Go internal version of many libraries. They also suggested moving the install of the frontend pieces to postInstall, rather than overriding the standard installPhase.

And finally @SuperSandro2000 reviewed with some naming changes to simplify the package definitions and reduce repetition. I also noticed that I was unnecessarily declaring the frontend definition using the rec keyword. Once these were fixed the PR was merged!

Contribution Tips #

I’m still relatively new to this - you can see that my list of contributions is not extensive, but I’ve been through the process of adding, updating and refining packages and modules a few times.

Take this advice for what it is, but I hope you find it helpful:

Read the contribution guidelines: This is probably the best start. Try to follow the guidance as best you can, and take a look at some other recently closed Pull Requests to see if you can pick up on any patterns.
Be humble: Everyone is new when they start. You might find some reviews a little terse. Rest assured that this is only because the maintainers are incredibly busy. Assume good intentions, remain polite and respectful. Nixpkgs is one of the busiest Github repositories I’ve seen, and it’s largely maintained by volunteers who often have limited time to give for a huge number of contributions.
Be patient: Don’t expect a review immediately. If you’re looking for a preliminary review, feel free to ping me on Mastodon and I’ll try to offer any guidance I can.
Use the Discourse: Because of the volume of PRs, the community run ongoing “PRs Ready for Review” threads on Discourse which you can use to attract reviewers if your PR has been stale a while. It’s also a great place to ask for help.

Once merged, commits trickle through the various branches of the repository. You can track the progress of your PR in that process once it’s merged using the PR Tracker. You can see an example of this by tracking the Scrutiny PR:

Summary & Thanks #

My Scrutiny packages, module and tests landed in nixpkgs approximately 4 days after I submitted the pull request. As ever, I received a bunch of helpful reviews from the community, and Scrutiny is now easier than ever to consume on NixOS.

You can see the finished components here:

If you were previously using my flake to consume Scrutiny, you’ll now see a deprecation warning. To migrate over, import the module from unstable in your flake and ensure that you explicitly enable the collector:

1
2
3
4
5
6


{
 services = {
 scrutiny.enable = true;
 scrutiny.collector.enable = true;
 };
}

Before I wrap up, I’d like to thank all those people who took the time to review the PR, and especially @JulienMalka and @RaitoBezarius who have helped me through countless contributions over the past months and been incredibly responsive to my questions.

Give it a go and let me know how you get on!

Packaging Scrutiny for NixOS

Fri, 16 Feb 2024 00:00:00 +0000

Update: Since writing this post, I’ve contributed Scrutiny upstream to nixpkgs/NixOS. You can see the write up of that effort here.

Introduction #

In a recent (well, recent-ish) episode of the Self Hosted Show, there was some talk of a hard drive monitoring tool called Scrutiny. Scrutiny is a hard drive monitoring tool that exposes S.M.A.R.T data in a nice, clean dashboard. It gathers that S.M.A.R.T data using the venerable smartd, which is a Linux daemon that monitors S.M.A.R.T data from a huge number of ATA, IDE, SCSI-3 drives. The code is available on Github.

The aim of running such monitoring is to detect and replace failing hard drives before they cause an outage, or any data loss. Depending on the firmware of a drive, there are potentially hundreds of S.M.A.R.T attributes that can be collected, so it can be hard to understand which to pay attention to.

As good as smartd is, it has some shortcomings such as not recording historical data for each attribute, and relying upon value thresholds that are set by the manufacturers of the drive. Scrutiny aims to provide historical tracking of attribute readings, and thresholds based on real-world data from Backblaze.

This all looked very compelling, but it hasn’t been packaged for NixOS as far as I can tell. This post is quite “code heavy” and is intended as a bit of a deep-dive/walkthrough for people who are new to packaging in Nix. I’m certainly no expert here, and if you spot something I’ve done wrong - I’d love to hear about it!

Below is a screenshot of Scrutiny’s dashboard:

Architecture #

In order to get Scrutiny up and running, there were a few components that all needed packaging separately.

Scrutiny itself is made up of the following:

A web application backend - written in Go
A web application frontend - written in NodeJs/Angular
A collector service - written in Go

The web application is the dashboard that you’ll see the pretty screenshots of. The collector service is designed to be run on an interval to collect information from smartd, and send it to the web application’s API.

Scrutiny relies upon InfluxDB to store data about smart attributes in a timeseries.

Thinking about how to structure things, I decided that I would build two separate Nix packages: one for the dashboard and UI, and another for the collector. It seems one could run the collector and dashboard on different machines, so this seemed like a logical split.

Packaging for NixOS #

Packaging the Dashboard #

I decided to start with the web application. Because of how the project is laid out, this means combining two separate derivations based built from the same source code: one for the UI and one for the backend. I began by creating a file that would carry common attributes for each of the derivations, such as the version, repository information, hash, etc.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


# common.nix
{ pkgs , ... }:
let
 name = "scrutiny";
 version = "0.7.2";
in
{
 inherit name version;

 # Fetch the source code from Github, referring to
 # the version required by Git tag.
 repo = pkgs.fetchFromGitHub {
 owner = "AnalogJ";
 repo = name;
 rev = "v${version}";
 sha256 = "sha256-UYKi+WTsasUaE6irzMAHr66k7wXyec8FXc8AWjEk0qs=";
 };

 # ...
}

I then started packaging the frontend:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36


{ pkgs, ... }: {
 # Import some common attributes such
 # as name, version, src repo
 common = import ./common.nix { inherit pkgs; };
 inherit (common) name version repo vendorHash;

 # Create a package for the Javascript frontend
 frontend = pkgs.buildNpmPackage rec {
 inherit version;
 pname = "${name}-webapp-frontend";
 src = "${repo}/webapp/frontend";

 # This hash is generated at build time, and uniquely identifies
 # the cache of NodeJS dependencies used for the build.
 npmDepsHash = "sha256-M8P41LPg7oJ/C9abDuNM5Mn+OO0zK56CKi2BwLxv8oQ=";

 # Override the build phase to match the
 # upstream build process.
 buildPhase = ''
 runHook preBuild
 mkdir dist
 npm run build:prod --offline -- --output-path=dist
 runHook postBuild
 '';

 # Copy the output of the compiled javascript bundle
 # and site assets to the package output.
 installPhase = ''
 runHook preInstall
 mkdir $out
 cp -r dist/* $out
 runHook postInstall
 '';
 };
 # ...
}

This is a relatively simple derivation, mostly thanks to the magic of the buildNpmPackage function. This helper function takes care of creating an offline cache containing all of the NodeJS dependencies required to build the frontend. Nix builds are always done in an offline environment to help ensure reproducibility, so source code and dependencies are fetched (and hashed) early in the process before any software is actually built.

I chose to override the build phase to match the process used by the upstream Makefile. The result of this derivation is a package named scrutiny-webapp-frontend, which contains just the built output from the npm run build:prod command.

Next up was the dashboard backend. Another pleasingly simple derivation thanks to some help from buildGoModule:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


pkgs.buildGoModule rec {
 # The vendor hash is used for multiple packages, and
 # thus centralised and imported here.
 inherit version vendorHash;
 pname = "${name}-webapp-backend";

 # Use the source block defined in 'common.nix'
 src = repo;

 CGO_ENABLED = 0;

 # Override the build phase to ensure the correct
 # binary is built. This project ships both Go applications
 # and the NodeJS frontend in the same repository, so some
 # specificity is required.
 buildPhase = ''
 runHook preBuild
 go build \
 -o scrutiny-web \
 -ldflags="-extldflags=-static" \
 -tags "static netgo" \
 ./webapp/backend/cmd/scrutiny
 runHook postBuild
 '';

 # Add the built binary to the package output, as well
 # as the build output from the frontend.
 installPhase = ''
 mkdir -p $out/bin $out/share/scrutiny
 cp scrutiny-web $out/bin/scrutiny-web
 cp -r ${frontend}/* $out/share/scrutiny
 '';
}

This one is a little more interesting. There are some commonalities, such as setting the vendorHash to ensure the correct Go dependencies are used (imported from common.nix in this case), and overriding the build to match the upstream process and ensure the right binary is built.

Where things differ is in the install phase, where the contents of the scrutiny-webapp-frontend derivation is copied into the output of this derivation - which will ultimately result in a single Nix package (named scrutiny-webapp-backend) which will contain both the frontend and backend components of the application. This is ultimately exposed in an overlay as a package simply named scrutiny.

You can see the finished product (with additional package metadata) in app.nix and common.nix on Github.

Packaging the Collector #

The collector is just a single, statically compiled Go binary, and as such the derivation very much resembles that of the web application backend above:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39


{ pkgs, lib, ... }:
let
 # Again import common attributes from 'common.nix'
 common = import ./common.nix { inherit pkgs; };
 inherit (common) name version repo vendorHash;
in
pkgs.buildGoModule rec {
 inherit version vendorHash;
 pname = "${name}-collector";
 src = repo;

 # We'll create a wrapper script for the
 # collector, which makeWrapper helps with.
 buildInputs = with pkgs; [ makeWrapper ];

 CGO_ENABLED = 0;

 # Override the build phase to ensure the correct
 # binary is built.
 buildPhase = ''
 runHook preBuild
 go build \
 -o scrutiny-collector-metrics \
 -ldflags="-extldflags=-static" \
 -tags "static netgo" \
 ./collector/cmd/collector-metrics
 runHook postBuild
 '';

 # Install both the binary, and a generated wrapper script
 # which ensures that 'smartctl' is in the PATH of the collector.
 installPhase = ''
 mkdir -p $out/bin
 cp scrutiny-collector-metrics $out/bin/scrutiny-collector-metrics

 wrapProgram $out/bin/scrutiny-collector-metrics \
 --prefix PATH : ${lib.makeBinPath [ pkgs.smartmontools ]}
 '';
}

Of interest here is the installPhase. The collector works by invoking smartctl to scrape information from smartd. Scrutiny itself expects that tool to be readily available in it’s PATH, and to accomplish that I used the makeWrapper package to create a wrapper script that ensures scrutiny-collector-metrics is executed with the PATH correctly set such that smartctl can be found.

The final derivation for the collector can be seen in collector.nix on Github.

Writing a NixOS Module #

While I now had functioning packages for all of Scrutiny’s components, for them to function correctly on a machine there are a few things that need to be in place:

The dashboard package must be installed and started
The collector collector package must be installed and started
InfluxDB must be installed and started
Scrutiny dashboard must be configured to speak to the host’s InfluxDB
smartd must be installed and running

This sort of challenge is exactly what the NixOS modules system aims to solve. Modules take a set of configuration attributes, and convert them into rendered system configuration in the form of systemd units, configuration files and more.

It’s tempting to try to support all possible configurations when writing a module, but I generally prefer to start small, and support only the configuration I need for my use-case. This keeps things easy to test (and leaves some fun for the future!). In this case, the module would be first installed on a server which hosts a set of services behind the Traefik reverse proxy. To work out what configuration I wanted to provide, I looked at the upstream’s example config file. The important things that stood out to me for consideration were:

The location of the web UI files to serve
The host/port of the InfluxDB instance
The “base path” - Scrutiny will be exposed at https://<tailscale node>/scrutiny

As mentioned before - each NixOS module consists of some options, and some rendered config. The options block for my Scrutiny web app looks like the below snippet. I’ve omitted comments this time, as I think the language is quite descriptive here:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48


{
 let
 cfg = config.services.scrutiny;
 in
 options = {
 services.scrutiny = {
 enable = lib.mkEnableOption "Enables the scrutiny web application";

 # Use the `scrutiny` package by default.
 package = lib.mkPackageOptionMD pkgs "scrutiny" { };

 port = lib.mkOption {
 type = lib.types.port;
 default = 8080;
 description = lib.mdDoc "Port for web application to listen on";
 };

 host = lib.mkOption {
 type = lib.types.str;
 default = "0.0.0.0";
 description = lib.mdDoc "Interface address for web application to bind to";
 };

 basepath = lib.mkOption {
 type = lib.types.str;
 default = "";
 description = lib.mdDoc ''
 If Scrutiny will be behind a path prefixed reverse proxy, you can override this
 value to serve Scrutiny on a subpath.

 Do not include the leading '/'.
 '';
 };

 openFirewall = lib.mkOption {
 type = lib.types.bool;
 default = false;
 description = "Open the default ports in the firewall for Scrutiny.";
 };

 logLevel = lib.mkOption {
 type = lib.types.enum [ "INFO" "DEBUG" ];
 default = "INFO";
 description = lib.mdDoc "Log level for Scrutiny.";
 };
 };
 };
}

This provides some basic configuration for the attributes I care about - noticeably missing is any advanced configuration for InfluxDB (such as org, bucket, token), and and any ability to configure notifications which Scrutiny supports through the excellent shoutrrr library. These things will come later.

I needed a convenient way to convert this Nix-native configuration format into the right format for Scrutiny - I wrote a small Nix function to help with that, which takes configuration elements from the options defined above, and writes them into a small YAML file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


{
 mkScrutinyCfg = cfg: pkgs.writeTextFile {
 name = "scrutiny.yaml";
 text = ''
 version: 1
 web:
 listen:
 port: ${builtins.toString cfg.port}
 host: "${cfg.host}"
 basepath: "${if cfg.basepath != "" then "/" + cfg.basepath else ""}"

 database:
 location: "/var/lib/scrutiny/scrutiny.db"

 src:
 frontend:
 path: "${pkgs.scrutiny}/share/scrutiny"

 log:
 level: "${cfg.logLevel}"
 '';
 };
}

Note that this hard-codes some key elements, such as the path of the web application assets which are shipped as part of the scrutiny package.

Let’s take a look at the part of the module which turns these options into a rendered system configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


{
 config = {
 # If scrutiny is enabled, also enable InfluxDB with default settings
 services.influxdb2 = lib.mkIf cfg.enable {
 enable = true;
 };

 # Open the relevant ports in the system firewall if configured
 networking.firewall = lib.mkIf cfg.openFirewall {
 allowedTCPPorts = [ cfg.port ];
 };

 # If Scrutiny is enabled, create a systemd unit to start it
 systemd = {
 services = {
 scrutiny = lib.mkIf cfg.enable {
 description = "Hard Drive S.M.A.R.T Monitoring, Historical Trends & Real World Failure Thresholds";
 wantedBy = [ "multi-user.target" ];
 after = [ "network.target" ];
 serviceConfig = {
 # Don't run the application a root - create a dynamic user as it starts
 DynamicUser = true;
 # Start the application with a config rendered by the helper function
 ExecStart = "${cfg.package}/bin/scrutiny-web start --config ${mkScrutinyCfg cfg}";
 Restart = "always";
 StateDirectory = "scrutiny";
 StateDirectoryMode = "0750";
 };
 };
 };
 };
 };
}

That’s enough to get the dashboard started, but it doesn’t take care of starting the collector. For that, I added a couple more configuration options to the module:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


{
 let
 collectorCfg = config.services.scrutiny.collector;
 in
 {
 options = {
 services.scrutiny = {
 # ...
 collector = {
 enable = lib.mkOption {
 type = lib.types.bool;
 default = cfg.enable;
 description = lib.mdDoc "Enables the scrutiny collector";
 };

 # Use the `scrutiny-collector` package by default.
 package = lib.mkPackageOptionMD pkgs "scrutiny-collector" { };

 endpoint = lib.mkOption {
 type = lib.types.str;
 default = "http://${cfg.host}:${builtins.toString cfg.port}/${cfg.basepath}";
 description = lib.mdDoc "Scrutiny app API endpoint for sending metrics to.";
 };

 interval = lib.mkOption {
 type = lib.types.str;
 default = "15m";
 description = lib.mdDoc ''
 Interval on which to collect information about disks.

 Examples: 15m, 10s, 2h.
 '';
 };
 };
 };
 };
 }
}

With that configuration in place, I needed to adjust the rendered configuration to include starting the collector. The collector is designed to run on an interval to post metrics to the dashboard. The upstream achieves this with cron, but I decided to use systemd timers:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39


{
 config = {
 # ...
 # If the collector is enabled, enable smartd so that the
 # collector can scrape metrics from it
 services.smartd = {
 enable = collectorCfg.enable;
 extraOptions = [ "-A /var/log/smartd/" "--interval=600" ];
 };

 systemd = {
 services = {
 # Setup a systemd unit to start the collector
 scrutiny-collector = lib.mkIf collectorCfg.enable {
 description = "Scrutiny Collector Service";
 environment = {
 COLLECTOR_API_ENDPOINT = "${collectorCfg.endpoint}";
 };
 serviceConfig = {
 Type = "oneshot";
 ExecStart = "${cfg.collector.package}/bin/scrutiny-collector-metrics run";
 };
 };
 };

 # Set up a systemd timer to trigger the collector at an
 # interval (default 15m).
 timers = {
 scrutiny-collector = lib.mkIf collectorCfg.enable {
 timerConfig = {
 OnBootSec = "5m";
 OnUnitActiveSec = "${collectorCfg.interval}";
 Unit = "scrutiny-collector.service";
 };
 };
 };
 };
 };
}

And that’s it! You can see the final module all stitched together on Github.

Automated Testing #

To me, a super exciting part of the NixOS ecosystem is its automated testing framework, which is used to great effect for testing hundreds of packages end-to-end before they’re released to the various NixOS channels. The NixOS testing framework provides a set of helpers for spawning fresh NixOS virtual machines, and ensuring that applications can start, services have the right side effects, etc.

I wanted to write a simple test to validate that Scrutiny will continue to work as I update my flake. In my view, the test needed to:

Ensure that the packages could be built
Ensure that when services.scrutiny.enable = true is set, the services start
Ensure that the dashboard app renders the UI
Ensure that the metrics collector can speak to the dashboard

Most of these are relatively trivial - one piece that’s a little harder is testing that the UI renders correctly. The application is rendered using Javascript client-side, so a simple curl won’t get us the results we’re expecting. I had previously used selenium for this purpose when I submitted a test for the LXD UI, so I chose to use that approach again.

The test code can be seen below, or on Github:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69


{
 name = "scrutiny";
 nodes = {
 # Configure a NixOS virtual machine for the test
 machine = { self, pkgs, lib, ... }: {
 # Ensure that my NixOS module is imported
 imports = [ self.nixosModules.scrutiny ];
 # Add the package overlay from my flake so that `pkgs.scrutiny` resolves.
 nixpkgs.overlays = [ self.outputs.overlays.additions ];

 # Configure the VM to use my module and enable it
 services.scrutiny.enable = true;


 environment.systemPackages =
 let
 # A selenium script that fetches the dashboard using geckodriver
 seleniumScript = pkgs.writers.writePython3Bin "selenium-script"
 {
 libraries = with pkgs.python3Packages; [ selenium ];
 } ''
 from selenium import webdriver
 from selenium.webdriver.common.by import By
 from selenium.webdriver.firefox.options import Options
 from selenium.webdriver.support.ui import WebDriverWait

 options = Options()
 options.add_argument("--headless")
 service = webdriver.FirefoxService(executable_path="${lib.getExe pkgs.geckodriver}") # noqa: E501

 driver = webdriver.Firefox(options=options, service=service)
 driver.implicitly_wait(10)
 driver.get("http://localhost:8080/web/dashboard")

 wait = WebDriverWait(driver, 60)

 # Look for some elements that should be rendered by the Javascript bundle in the UI
 assert len(driver.find_elements(By.CLASS_NAME, "mat-button-wrapper")) > 0
 assert len(driver.find_elements(By.CLASS_NAME, "top-bar")) > 0

 driver.close()
 '';
 in
 with pkgs; [ curl firefox-unwrapped geckodriver seleniumScript ];
 };
 };
 # This is the test code that will check if our service is running correctly:
 testScript = ''
 start_all()

 # Wait for InfluxDB to be available
 machine.wait_for_unit("influxdb2")
 machine.wait_for_open_port(8086)

 # Wait for Scrutiny to be available
 machine.wait_for_unit("scrutiny")
 machine.wait_for_open_port(8080)

 # Ensure the API responds as we expect
 output = machine.succeed("curl localhost:8080/api/health")
 assert output == '{"success":true}'

 # Start the collector service to send some metrics
 collect = machine.succeed("systemctl start scrutiny-collector.service")

 # Ensure the application is actually rendered by the Javascript
 machine.succeed("PYTHONUNBUFFERED=1 selenium-script")
 '';
}

This relatively short code snippet will take care of:

Building a dedicated virtual machine according to the machine spec
Starting the VM
Running the tests inside the VM (specified in `testScript)
Collecting the results

I ended up adding this check in a Github Actions workflow so that it’s run each time I make a change to my flake.

Try It! #

You can give this a go too! If you’ve got a NixOS machine, or perhaps a virtual machine you’ve been experimenting with, then you need only add the following to your flake:

1
2
3
4
5
6
7
8
9


{
 inputs = {
 nixpkgs-unstable.url = "github:nixos/nixpkgs/nixos-unstable";
 jnsgruk.url = "github:jnsgruk/nixos-config";
 jnsgruk.inputs.nixpkgs.follows = "nixpkgs-unstable";
 # ...
 }
 # ...
}

And in your NixOS machine configuration:

1
2
3
4
5
6


# Where 'inputs' is the inputs attribute of your flake
{ inputs, ... }: {
 imports = [ inputs.jnsgruk.nixosModules.scrutiny ];
 nixpkgs.overlays = [ inputs.jnsgruk.overlays.additions ];
 services.scrutiny.enable = true;
}

The next time you rebuild your system, you should be able to browse to http://localhost:8080 and see some hard drive metrics!

What’s Next? #

This was a pretty quick exercise - I’d estimate at about 3 hours in total. No doubt I will find some bugs, but already in my mind are the following future improvements:

Add some more tests that exercise more of the config options
Enable notifications configuration in the module
Submit the packages/module to the upstream or to nixpkgs

I also need to go and investigate this rather sad looking hard disk…

That’s all for now! Thanks for reading if you got this far!

Integration testing with NixOS in Github Actions

Sat, 10 Feb 2024 00:00:00 +0000

Introduction #

While in my own time I’ve tended toward NixOS over the past 18 months, in my day-to-day work for Canonical I’m required to interact with a fair few of our products - and particularly build tools.

I frequently need to use some combination of Snaps, Charms and Rocks. Each of these have their own “craft” build tools (snapcraft, charmcraft and rockcraft), which are distributed exclusively as Snap packages and thus a little tricky to consume from NixOS.

The Problem #

Packaging the tools for Nix was a little repetitive, but not particularly difficult. They’re all built with Python, and share a common set of libraries. Testing that the packages were working correctly (i.e. could actually build software) on NixOS using the NixOS version of LXD in Github Actions proved more difficult.

Github Actions defaults to Ubuntu as the operating system for its runners - an entirely sensible choice, but not one that was going to help me test packages could work together on NixOS.

I could have hosted my own Github Actions runners to solve the problem, but I didn’t want to maintain such a deployment.

For a while I relied on just testing each of the crafts locally before pushing, and the CI simply installed the Nix package manager on the runners (using the excellent Nix installer from Determinate Systems) and ensured that the build could succeed, but this left a lot to be desired - particularly when I accidentally (and somewhat inevitably) broke one of the packages.

KVM for Github Actions #

Some time later I came across this post on the Github Blog, stating the following:

Starting on February 23, 2023, Actions users […] will be able to make use of hardware acceleration […].

What follows is an example of a relatively simple addition to a Github Workflow to enable KVM on Github Actions runners:

1
2
3
4
5


- name: Enable KVM group perms
 run: |
 echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules
 sudo udevadm control --reload-rules
 sudo udevadm trigger --name-match=kvm

Given the ability to relatively easily create NixOS VMs from a machine configuration, this should enable me to run a NixOS VM inside my Github Actions runners, and use that VM to run end to end tests of my craft packages.

After some quick tests, I confirmed that the above snippet worked just fine on the freely available runners that are assigned to public projects. After tooting excitedly about this, it was also picked up by the folks at Determinate Systems who promptly added support for this in their Nix install Github Action - enabling the feature by default.

Building VMs with Nix #

A really nice feature of NixOS that I discovered relatively late, is that given a NixOS machine configuration it’s trivial to build a virtual machine image for that configuration. This has the nice property that one can actually boot a VM-equivalent of any previously defined machines. You could, for example, boot a VM-equivalent of my laptop with the following command:

1

nix run github:jnsgruk/nixos-config#nixosConfigurations.freyja.config.system.build.vm

In order to test my craft tools, I needed a relatively simple NixOS VM that had LXD enabled, and my craft tools installed. My test VM configuration looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45


{ modulesPath, flake, pkgs, ... }: {
 # A nice helper that handles creating the VM launch script, which in turn
 # ensures the disk image is created as required, and QEMU is launched
 # with sensible parameters.
 imports = [ "${modulesPath}/virtualisation/qemu-vm.nix" ];

 # Define the version of NixOS and the architecture.
 system.stateVersion = "23.11";
 nixpkgs.hostPlatform = "x86_64-linux";

 # This overlay is provided by the crafts-flake, and ensures that
 # 'pkgs.snapcraft', 'pkgs.charmcraft', 'pkgs.rockcraft' all resolve to
 # the packages in the flake.
 nixpkgs.overlays = [ flake.outputs.overlay ];

 # These values are tuned such that the VM performs on Github Actions runners.
 virtualisation = {
 forwardPorts = [{ from = "host"; host.port = 2222; guest.port = 22; }];
 cores = 2;
 memorySize = 5120;
 diskSize = 10240;
 };

 # Configure the root user without password and enable SSH.
 # This VM will only ever be used in short-lived testing environments with
 # no inbound networking permitted, so there is minimal (if any) risk.
 # If you put this VM on the internet, you can keep the pieces! :)
 networking.firewall.enable = false;
 services.openssh.enable = true;
 services.openssh.settings.PermitRootLogin = "yes";
 users.extraUsers.root.password = "password";

 # Ensure that LXD is installed, and started on boot.
 virtualisation.lxd.enable = true;

 # Include the `craft-test` script, ensuring the craft apps are installed
 # and included in its PATH.
 environment.systemPackages = [
 (pkgs.writeShellApplication {
 name = "craft-test";
 runtimeInputs = with pkgs; [ unixtools.xxd git snapcraft charmcraft rockcraft ];
 text = builtins.readFile ./craft-test;
 })
 ];
}

Anybody can build and launch this VM trivially:

1

nix run github:jnsgruk/crafts-flake#testVM

Writing a Github workflow #

All the building blocks are in place! I wanted to keep the actual workflow definition for the tests as clean and understandable as possible, so I put together the craft-test script as a small helper which automates the building of real artefacts. An example invocation might be:

1

bash craft-test snapcraft

On each invocation, the script creates temporary directory, clones some representative build files for the selected craft tool, and launches the craft. The repos it uses for the representative packages are hard-coded for each craft for now.

I wrote one more small helper script to simplify connecting to the VM with the required parameters. It’s a wrapper around ssh and sshpass that’s hard-coded with the credentials of the test VM (don’t @ me!), and executes commands over SSH in the test VM. Using this script, one can bash vm-exec -- craft-test snapcraft and the craft-test script will be executed over SSH in the VM.

With all that said and done, the resulting workflow is pleasingly simple:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


# ...
jobs:
 test:
 runs-on: ubuntu-latest
 strategy:
 matrix:
 package: ["charmcraft", "rockcraft", "snapcraft"]
 steps:
 - name: Checkout flake
 uses: actions/checkout@v4

 - name: Install nix
 uses: DeterminateSystems/nix-installer-action@v9

 - name: Build and run the test VM
 run: |
 nix run .#testVm -- -daemonize -display none

 - name: Test ${{ matrix.package }}
 run: |
 nix run .#testVmExec -- craft-test ${{ matrix.package }}

A separate job is run for each of the crafts, and a real artefact is built in each, giving reasonable confidence that the consumers of my flake will be successful when building snaps, rocks and charms natively on NixOS. A successful run can be seen here.

Summary #

In this article we’ve covered:

Enabling KVM on Github Runners
Building NixOS VMs using Flakes
Booting NixOS VMs in Github Actions

If you’d like to build snaps, rocks or charms and you’re running NixOS, you can run the tools individually from my flake:

1
2
3
4
5
6
7
8


# Run charmcraft
nix run github:jnsgruk/crafts-flake#charmcraft

# Run rockcraft
nix run github:jnsgruk/crafts-flake#rockcraft

# Run snapcraft
nix run github:jnsgruk/crafts-flake#snapcraft

Or you can check out the README for instructions on how to integrate into your Nix config using overlays!

That’s all for now! 🤓

Building a blog with Go, Nix and Hugo

Fri, 12 Jan 2024 00:00:00 +0000

Introduction #

I’ve been procrastinating about blogging for about a decade. Long-form writing is a format I enjoy consuming, and I’ve learned a huge amount from the various blogs I’ve subscribed to over the years. Yet, there have always been a couple of nagging reasons preventing me from starting my own:

Why would anyone want to read my blog?
How would I come up with ideas for content?
Where would I find the time to write the blog?

Perhaps what’s changed recently is a new found enjoyment in some side projects, and the realisation that I might just write about things for the love of it, whether or not its directly useful to anyone else. Of course I’d love people find the content useful, engage, etc., but that isn’t my primary motivation.

The second two points are closely linked, but I ultimately decided they didn’t matter. So I present this blog as a self-indulgence, and something that I’ll update when I’m excited about writing, and not feel bad about the rest of the time! 😉

Being a Software Engineer, I quickly established that it was important to spend time over-engineering my blog before sitting down and writing any content, and this first post illustrates that journey </sarcasm>.

Rendering the blog #

I’ve been fond of Hugo for years now (I even named my son Hugo 😉). I’ve used it in a few projects, and I find it to be largely easy to understand and well maintained. My previous site was built with Hugo, using a theme named congo which I’d been underutilising by only creating a “business card” style page. I decided to stick with this setup, and just use more of the layouts provided by the theme.

In many ways the Hugo site is the most “boring” part of the site. The source code is all available in the site directory of the Github repo, but I won’t talk much about the site itself in this blog, as there isn’t much more to say!

Serving the blog #

4 years ago, I made the first commit to a project named gosherve. This was one of my first adventures into Go, and I was left with a small, but functional web server that could serve files from a directory, and serve redirects specified in a publicly accessible text file.

I chose to host the redirect definitions in a Github Gist. When I want to share a link frequently, or place one somewhere visible like a slide, I update the Gist with a new alias, and jnsgr.uk/<alias> comes online as a handy short link the first time someone requests it.

Last year I decided to use gosherve as a tool for learning more about slog and the Go Prometheus client. I did some refactoring that tidied up the logging, and introduced basic metrics for the number of times each redirect was accessed, how many redirects were defined and the total number of redirects served.

For my new blog, I wanted to keep the short URLs I’d defined, and I wanted to embed the static site into the binary to make deployment as simple as possible. I made two small changes to gosherve to enable this:

The first change enables the server and logging components of gosherve to be imported as libraries, and the second enables gosherve to serve files from a filesystem (and critically, an embedded filesystem).

Embedding the blog #

One of the things I love about Go is how rich the standard library is, and how it can simplify the creation of small, but powerful applications. The code for my website’s server as I write this is below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40


package main

//go:generate hugo --minify -s site -d ../public

import (
 "embed"
 // ...
 "github.com/jnsgruk/gosherve/pkg/logging"
 "github.com/jnsgruk/gosherve/pkg/server"
)

var (
 commit string = "dev"
 logLevel = flag.String("log-level", "info", "log level of the application")
 redirectsURL = "https://gist.githubusercontent.com/jnsgruk/b590f114af1b041eeeab3e7f6e9851b7/raw"

 //go:embed public
 publicFS embed.FS
)

func main() {
 flag.Parse()
 logging.SetupLogger(*logLevel)

 fsys, err := fs.Sub(publicFS, "public")
 if err != nil {
 slog.Error(err.Error())
 os.Exit(1)
 }

 s := server.NewServer(&fsys, redirectsURL)

 err = s.RefreshRedirects()
 if err != nil {
 slog.Error("unable to fetch redirect map", "error", err.Error())
 os.Exit(1)
 }

 s.Start()
}

I’ve omitted some comments, imports and logging for brevity here, but the complete file (at 55 lines) can be found on Github for the curious.

There key elements here are:

//go:generate hugo --minify -s site -d ../public: this makes sure go generate invokes Hugo to build the site and place the output in the public directory.
//go:embed public: embeds the public directory into the binary as an embedded filesystem.

Building the blog #

For the last 18 months, I’ve been enjoying Nix and NixOS for my personal machines, so I wanted to use my newly acquired knowledge to package and build my website using Nix.

There is lots of ongoing discussion in the Nix community about Flakes, which are an experimental technology aimed at simplifying usability and improving reproducibility of Nix installations. There are lots of other facets to the discussion which I’ll likely touch upon in future posts, but for now I’ll just say that I like Flakes, and they were the obvious choice for packaging this site.

Packaging a Go application for Nix is relatively simple thanks to helpers like buildGoModule. I had to make some minor modifications to accommodate the go generate step to build the Hugo site, and patch out some elements of the Hugo site that relied upon access to the local Git tree, but the resulting derivation remains relatively easy to digest (see flake.nix):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34


buildGoModule {
 inherit version;
 pname = "jnsgruk";
 src = lib.cleanSource ./.;

 vendorHash = "sha256-4f04IS76JtH+I4Xpu6gF8JQSO3TM7p56mCs8BwyPo8U=";
 buildInputs = [ cacert ];
 nativeBuildInputs = [ hugo ];

 # Nix doesn't play well with Hugo's "GitInfo" module, so disable it and inject
 # the revision from the flake.
 postPatch = ''
 substituteInPlace ./site/layouts/shortcodes/gitinfo.html \
 --replace "{{ .Page.GitInfo.Hash }}" "${rev}"

 substituteInPlace ./site/config/_default/config.yaml \
 --replace "enableGitInfo: true" "enableGitInfo: false"
 '';

 # Generate the Hugo site before building the Go application which embeds the
 # built site.
 preBuild = ''
 go generate ./...
 '';

 ldflags = [ "-X main.commit=${rev}" ];

 # Rename the main executable in the output directory
 postInstall = ''
 mv $out/bin/jnsgr.uk $out/bin/jnsgruk
 '';

 meta.mainProgram = "jnsgruk";
};

This defines a Nix package named jnsgruk, containing a single binary at bin/jnsgruk. This binary can be run anywhere to get a working version of this site. You can even try at home with:

1

nix run github:jnsgruk/jnsgr.uk

Deploying the blog #

I’ve been hosting my site on Fly.io without issue for a couple of years. They have a nice feature that allows you to Deploy via Dockerfile, where their command-line utility flyctl will send off a local Dockerfile to be built on their infrastructure and then launched, and that had been working great in previous versions of my site.

I wanted to be able to build and run the exact same bits on my own machines as were hosted by Fly, so I opted to build an OCI image with Nix, then upload that to Fly.io’s registry as part of the deployment. Adding a container image to the flake as an additional output was simple:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


dockerTools.buildImage {
 name = "jnsgruk/jnsgr.uk";
 tag = version;
 created = "now";
 copyToRoot = buildEnv {
 name = "image-root";
 paths = [ self.packages.${system}.jnsgruk cacert ];
 pathsToLink = [ "/bin" "/etc/ssl/certs" ];
 };
 config = {
 Entrypoint = [ "${lib.getExe self.packages.${system}.jnsgruk}" ];
 Expose = [ 8080 8801 ];
 User = "10000:10000";
 };
};

All that remained was to wire up Github Actions to build and deploy the site each time I make a new commit. Because the build tooling setup is all handled by Nix, the resulting Github workflow is quite brief:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41


name: Fly Deploy
on:
 push:
 branches:
 - main

permissions:
 packages: write

jobs:
 deploy:
 name: Deploy app
 runs-on: ubuntu-latest
 steps:
 - name: Checkout
 uses: actions/checkout@v4

 - name: Install nix
 uses: DeterminateSystems/nix-installer-action@v9

 - name: Login to GitHub Container Registry
 uses: docker/login-action@v3
 with:
 registry: ghcr.io
 username: ${{ github.actor }}
 password: ${{ secrets.GITHUB_TOKEN }}

 - name: Build container
 run: nix build -L .#jnsgruk-container

 - name: Upload container to ghcr.io
 run: |
 docker load < result
 docker tag "jnsgruk/jnsgr.uk:$(git rev-parse --short HEAD)" "ghcr.io/jnsgruk/jnsgr.uk:$(git rev-parse --short HEAD)"
 docker push "ghcr.io/jnsgruk/jnsgr.uk:$(git rev-parse --short HEAD)"

 - name: Deploy site
 run: |
 nix run nixpkgs#flyctl -- deploy -i "ghcr.io/jnsgruk/jnsgr.uk:$(git rev-parse --short HEAD)"
 env:
 FLY_ACCESS_TOKEN: ${{ secrets.FLY_API_TOKEN }}

And that’s the end! You’re reading this article as a result of the above workflow succeeding.

Summary #

I’ve never had a blog before, but I’m looking forward to documenting some of my adventures in Linux, Software Engineering, Technical Leadership and more over the coming year. Thanks for reading!

Nix on Jon Seager

Ubuntu Summit 25.10: Personal Highlights

DOOM in Space #

Infrastructure-Wide Profiling of Nvidia CUDA #

Inference Snaps #

Nøughty Linux: Ubuntu’s Stability Meets Nixpkgs’ Freshness #

Are we stuck with the same Desktop UX forever? #

Honorable Mentions #

Conclusion #

Packaging the Multipass Flutter GUI for NixOS

Introduction #

Housekeeping #

Restructuring #

Using symlinkJoin #

Building the GUI #

Update Script #

Summary #

Experimenting with Rust, Nix, K6 and Parca

Introduction #

Learning Rust #

Rewriting gosherve in Rust #

Basic Load Testing with k6 #

Nix-ified Load Testing #

Defining Test VMs #

Wrapping k6 #

Automating Test Execution #

Initial Load Test Results #

Profiling gosherve with Parca #

Instrumenting gosherve #

Initial optimisations #

Reducing Allocations #

Profiling servy? #

Final Results & Comparisons #

Summary #

Hot Tub Monitoring with Home Assistant and ESPHome

Introduction #

Energy Usage Monitoring #

Temperature Sensor: Initial Solution #

Temperature Sensor: Homemade Solution #

Connecting the ESP32 #

ESPHome Support #

ESPHome Dashboard #

Creating a Firmware #

3D Printing an Enclosure #

Home Assistant Dashboard #

Summary #

Writing a Home Assistant Core Integration: Part 2

Introduction #

Development Setup #

Integration Basics #

File Structure #

manifest.json #

__init__.py #

config_flow.py #

coordinator.py #

climate.py #

Testing #

Docs & Brand #

Contribution Process #

Results #

Summary #

Writing a Home Assistant Core Integration: Part 1

Introduction #

Designing the library #

Upstream API #

Basic Requirements #

Outline design/experience #

Python tools/libraries #

uv #

pydantic #

ruff #

Implementation #

Client implementation #

Caching #

Testing, CI & Publishing #

Testing #

CI #

Publishing #

Contributing to nixpkgs #

Summary #

Using `symlinkJoin` #

Rewriting `gosherve` in Rust #

Basic Load Testing with `k6` #

Wrapping `k6` #

Profiling `gosherve` with Parca #

Instrumenting `gosherve` #

Profiling `servy`? #

`manifest.json` #

`init.py` #

`config_flow.py` #

`coordinator.py` #

`climate.py` #

`uv` #

`pydantic` #

`ruff` #

Contributing to `nixpkgs` #