Blog on Jon Seager

ntpd-rs: it's about time!

Thu, 26 Mar 2026 00:00:00 +0000

This article was originally posted on the Ubuntu Discourse, and is reposted here. I welcome comments and further discussion in that thread.

Introduction #

I am thrilled to announce the next target in our campaign to replace core system utilities with memory-safe Rust rewrites in Ubuntu. In upcoming releases, Ubuntu will be adopting ntpd-rs as the default time synchronization client and server, eventually replacing chrony, linuxptp and with any luck, gpsd for time syncing use-cases.

ntpd-rs is a full-featured implementation of the Network Time Protocol (NTP), written entirely in Rust. Maintained by the Trifecta Tech Foundation as part of Project Pendulum, ntpd-rs places a strong focus on security, stability, and memory safety.

To deliver on this goal, we’re building on our partnership with the Trifecta Tech Foundation who are behind sudo-rs, zlib-rs and more. We will be funding the Trifecta Tech Foundation to build new features, enhance security isolation, and ultimately deliver a unified, memory-safe time synchronization utility for the Linux ecosystem. This work meshes well with the Trifecta Tech Foundations goals to improve the security of time synchronization everywhere.

NTP, NTS, and PTP #

Before diving into the mechanics and reasoning behind the transition, I wanted to give some background on the protocols at play, and the problems we’re hoping to solve. Keeping accurate time is a critical system function, not least because it involves constant interaction with the internet and forms the basis for cryptographic verification in protocols such as Transport Layer Security (TLS).

NTP (Network Time Protocol) is the foundational protocol that most operating systems implement to accurately determine the current time from a network source.

NTS (Network Time Security) is to NTP what HTTPS is to HTTP. Historically, the Network Time Protocol was used unencrypted, like many of the early web protocols. NTS introduces cryptographic security to time synchronization, ensuring that bad actors cannot intercept or spoof time data. We already pushed to make NTS the default out-of-the-box in Ubuntu 25.10, which we accomplished by migrating away from ntpd to chrony as the default time-syncing implementation.

PTP (Precision Time Protocol) is used for systems that require sub-microsecond synchronization. While the precision offered by a standard NTP deployment is sufficient for general-purpose computing, PTP is often used for complex, specialized deployments like telecommunications networks, power grids, and automotive applications.

Proven at Scale #

Transitioning core utilities in Ubuntu comes with a responsibility to ensure that replacements are of high quality, resilient and offer something to the platform. We may be the first major Linux distribution to adopt ntpd-rs by default, but we aren’t the first to recognize the readiness of ntpd-rs - it has already been proven at scale by Let’s Encrypt.

While Let’s Encrypt’s core Certificate Authority software has always been written in memory-safe Go, their server operating systems and network infrastructure historically relied on memory-unsafe languages like C and C++, which routinely led to vulnerabilities requiring patching.

Following extensive development, ntpd-rs was deployed to Let’s Encrypt’s staging environment in April 2024, and rolled out to full production by June 2024, marking a major milestone for ntpd-rs.

The fact that one of the world’s most prolific and security-conscious certificate authorities trusts ntpd-rs to keep time across its fleet should provide us, and our enterprise customers, with tremendous confidence in its resilience and suitability.

A Single, Memory-Safe Utility for NTP and PTP #

We want to provide a single utility for configuring both NTP/NTS and Precision Time Protocol (PTP) on Linux. The Trifecta Tech Foundation is concurrently developing Statime, a memory-safe PTP implementation that delivers synchronization performance on par with linuxptp, but with the goal of being easier to configure and use.

The goal is to integrate Statime’s PTP capabilities directly into ntpd-rs, improving the user experience by bringing all time synchronization concerns into one utility with common configuration and usage patterns, obviating the need for complex manual configuration (and troubleshooting) that users of linuxptp may be familiar with.

Timelines and Goals #

As with our transition to sudo-rs and uutils coreutils, leading the mainstream adoption of foundational system utilities comes with responsibility. We want to ensure that ntpd-rs matches the security isolation and performance standards our users expect from chrony.

Canonical is funding the Trifecta Tech Foundation’s development efforts toward these goals over the coming cycles. This work will take place between July 2026 and January 2027 in several major milestones. Our current timeline and targeted goals are:

Ubuntu 26.10: If all goes well, we aim to land the latest version of ntpd-rs in the archive, making it available to test.
Ubuntu 27.04: By 27.04, ntpd-rs should have integrated statime, and we will ship the unified client/server binary for NTP, NTS and PTP in Ubuntu by default, with the aim of providing a smooth migration path for those who already manage complex chrony configs.

To get us there, the Trifecta Tech Foundation will be working on the following items:

Feature Parity & Hardware Support: Adding gpsd IP socket support, multi-threading support for NTP servers, and support for multi-homed servers.
Security & Isolation: chrony is isolated via AppArmor and seccomp. We’ll be working on robust AppArmor and seccomp profiles for ntpd-rs to ensure we don’t buy memory safety at the cost of system-level privilege boundaries. We are also ensuring rustls can use openssl as a crypto provider to satisfy strict corporate cryptography policies.
PTP & Automotive Profiles: Adding support for gPTP, which will allow us to support complex deployments like the Automotive profile directly from nptd-rs (via Statime). Additionally, experimental support for the proposed Client-Server PTP protocol (CSPTP, IEEE 1588.1) will be added.
Benchmarking & Testing: Comprehensive benchmarking of long-term memory, CPU usage, and synchronization performance against chrony to give our cloud partners and enterprise users complete confidence in the transition.
User-experience: Logging improvements and enhancements to configuration that help users configure the time synchronisation target to optimise network usage, as well as improvements to the ntp-cli

About the Trifecta Tech Foundation #

Trifecta Tech Foundation is a non-profit and a Public Benefit Organisation (501(c)(3) equivalent) that creates open-source building blocks for critical infrastructure software. Their initiatives on data compression, time synchronization, and privilege boundary, impact the digital security of millions of people. If you’d like to support their work, please contact them via https://trifectatech.org/support.

Summary #

I am really excited to deepen our already productive relationship with the Trifecta Tech Foundation to make these transitions viable for the wider ecosystem. We’ll be working hard on testing and integration to ensure seamless migration paths, and heavily document the changes ahead of the 26.10 and 27.04 releases.

Stay tuned!

Brewlog: Coffee & Agents

Tue, 10 Mar 2026 00:00:00 +0000

Introduction #

I’m really into speciality coffee. In July 2025 I started tracking my coffee habits with Roastguide, a delightfully designed iOS app for people with a similar obsession to mine. It does a good job of tracking brews, roasters and bags, but over time I found myself wanting the data in a system that I controlled. I wanted to be able to query it, back it up, and adjust some of the flows to work better for how I brew and consume coffee.

Despite my past scepticism of the previous generation of coding tools, recent developments have made them hard to ignore, and I wanted an excuse to build something substantial from scratch to get some experience.

So I built b{rew}log, which is a self-hosted coffee logging platform. It tracks roasters, roasts, bags, brews, equipment, and cafe visits. It’s live at coffee.jnsgr.uk if you want to poke around and witness the depths of my strange filter coffee obsession!

This was by far the most complex project I’d built with Claude Code, or agentic coding tools in general. I’m responsible for the technology choices, architecture, and visual design, but I wrote almost none of the code myself.

This post will cover the core features of Brewlog and how I designed the app, and finish with some observations and tips about agentic programming.

Core Features #

Brewlog tracks roasters, roasts, bags, brews, equipment, and cafe visits. The landing page you see below summarises details of my last few brews, currently open bags of coffee and how much remains in each and some basic stats. When logged in, it provides quick controls to repeat a brew, or brew a particular coffee:

Brews are coffees I’ve made myself. Each brew is logged against a specific bag and captures the recipe (dose, water, time, temperature), the equipment used, and tasting notes. Over time this will build up a history of what I’ve brewed, how I brewed it and how much I use the brewing gear I’ve accumulated over time. The image below shows the detail page for a specific brew:

Finally, brewlog tracks cafes I’ve visited, and “cups”, which are coffees I’ve enjoyed, but not brewed myself.

The most fun part is the extensive stats page, which shows an interactive map of where my coffees are grown, roasted and drunk, as well as common flavour notes and brew times.

LLM-Powered Bag Scanning #

Roastguide has a nice feature that allows you to take a picture of a bag of coffee and it’ll fetch the details. If my understanding is correct, their implementation relies on a database of roasters and coffees that the creators maintain. My app wouldn’t have access to that database, nor did I want to spend too much time building ingestion pipelines to store lots of data about all the possible coffees/roasters.

In brewlog, I implemented this feature using LLM extraction: I take a photo and send it to a multi-modal model hosted by OpenRouter. The prompt instructs the model to extract text from the image, perform a web search, and return a JSON object containing details of the coffee or roaster. OpenRouter enables me to switch models easily without worrying about API differences.

In practice this works surprisingly well. Most speciality coffee bags are covered in useful information, and vision models are good at reading it. The main failure modes I see are mixing up roast names from the web and occasionally inventing tasting notes, but the process includes a review step which makes those cheap to correct.

The cost is almost negligible per scan. I’ve been using google/gemini-2.5-flash for the LLM extraction feature, which results in a cost of around $0.01 - $0.02 per scan.

Design Decisions #

Backend #

Brewlog is built with Rust and Axum for the backend. I’ve been learning Rust for a while now, and wanted to push further with a more substantial project. Templates are handled by Askama, whose compile‑time templates worked well with Claude because template errors surface as Rust compiler errors, which are picked up and fixed by the agent automatically.

The database is SQLite. A single file, easy to back up, easy to move around. For a single-user application like this, SQLite is more than sufficient and removes the need for a separate database server. Migrations are embedded in the binary and run automatically on application startup.

Frontend #

I wanted the app to feel modern, but keep much of the rendering server-side. I didn’t want to be serving a large client-side Javascript single-page application. I’ve been curious about the likes of HTMX, and the observations made in a post from last year on building apps with Eta, HTMX and Lit really resonated with me.

More recently I discovered Datastar, which has similar goals to HTMX, but it’s newer and results in cleaner, simpler templates. Where HTMX swaps HTML fragments, Datastar adds a reactive signal system and can patch both HTML and JSON data from the server.

This was a bit of a test for Claude Code, since Datastar was new enough that it likely didn’t feature in the training data for the model, but its ability to read and digest the documentation was quite startling.

While occasionally the agent regressed to vanilla fetch calls, or manual Javascript DOM manipulation, I treated that as a hole in my instructions, not the model. Each time it happened I updated CLAUDE.md to reinforce the Datastar patterns and prevent the agent making the same mistake again.

Single User, Passkeys Only #

Brewlog is deliberately single-user. I’m happy for people to self-host their own instance, but I don’t have much interest in providing a service more widely. The codebase is structured such that making it multi-user wouldn’t be a particularly complex task, but I’m not convinced I’ll ever do it.

Perhaps unusually, authentication via username and password is not supported. Authentication is only supported using passkeys via WebAuthn or with an API token.

Passkeys are phishing-resistant and significantly more difficult to brute-force both from an academic and a practical standpoint, which removes an entire class of abuse vector for my service.

Comprehensive CLI Client #

One of the many things I’ve learned from @niemeyer during my time at Canonical is to ensure that when new API endpoints are added to such applications, a corresponding CLI command or flag lands at the same time.

He also illustrated to me the power of shipping a single binary that is both a server, and a client to itself. The brewlog binary runs the server that hosts the API and web UI, but also serves as a first-class API client for the app.

If you’re building an API‑backed app, I highly recommend this pattern. It keeps the surface area small and gives you (and an agent…) a first‑class, scriptable way to exercise the API and troubleshoot problems in data transformation.

Domain-Driven Design #

The codebase follows a domain-driven design approach with four layers: domain (business logic only, no external dependencies), infrastructure (database, HTTP clients, third-party APIs), application (HTTP server, routes, middleware, services), and presentation (CLI commands and web view models):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36


┌─────────────────────────────────────────────────────────────┐
│ Presentation │
│ ┌─────────────────────────┐ ┌──────────────────────────┐ │
│ │ CLI │ │ Web │ │
│ │ roasters, roasts, bags │ │ views, templates │ │
│ │ brews, cups, gear ... │ │ roasters, roasts, bags │ │
│ └─────────────────────────┘ └──────────────────────────┘ │
├─────────────────────────────────────────────────────────────┤
│ Application │
│ ┌──────────────┐ ┌──────────────────┐ ┌─────────────────┐ │
│ │ Routes │ │ Services │ │ Server / State │ │
│ │ /api/* │ │ BagService │ │ Axum router │ │
│ │ /app/* │ │ BrewService │ │ AppState (DI) │ │
│ │ │ │ RoastService .. │ │ │ │
│ └──────────────┘ └──────────────────┘ └─────────────────┘ │
├─────────────────────────────────────────────────────────────┤
│ Domain (pure Rust — no framework dependencies) │
│ ┌──────────────────────┐ ┌─────────────────────────────┐ │
│ │ Entities & Values │ │ Repository Traits │ │
│ │ coffee/ roasters, │ │ trait RoasterRepository │ │
│ │ roasts, bags, │ │ trait RoastRepository │ │
│ │ brews, cups, gear │ │ trait BagRepository ... │ │
│ │ auth/ users, │ │ │ │
│ │ sessions, tokens │ │ Errors, IDs, Listing, │ │
│ │ analytics/ │ │ Formatting │ │
│ └──────────────────────┘ └─────────────────────────────┘ │
├──────────────────────────────────┬──────────────────────────┤
│ Infrastructure │ │
│ ┌────────────────────────────┐ │ ┌───────────────────┐ │
│ │ Repositories (SQLite) │ │ │ External Clients │ │
│ │ SqlRoasterRepository │ │ │ OpenRouter (AI) │ │
│ │ SqlRoastRepository │ │ │ Foursquare │ │
│ │ SqlBagRepository ... │ │ │ WebAuthn │ │
│ │ (implement domain traits) │ │ │ Backup │ │
│ └────────────────────────────┘ │ └───────────────────┘ │
└──────────────────────────────────┴──────────────────────────┘

In this model, dependencies flow inward. The domain layer knows nothing about Axum, SQLite, or any other implementation detail. Repository traits are defined in the domain and implemented in the infrastructure layer. This creates flexibility in the future if, for example, I want to move to PostgreSQL rather than SQLite, or if I want to change how I store images, etc.

It also simplifies testing. The domain-driven design approach encourages loose coupling and practices like dependency injection, which usually lead to simpler integration tests. For example, I could ask the agent for focused tests against pure domain logic without pulling in Axum or SQLite.

Agentic Coding Patterns #

A big motivation for this project was to learn some new technology and skills, and it certainly did that, but it also reinforced some patterns I’ve been using for a while now.

Pre-commit hooks as a contract #

I’ve never been too fond of pre-commit hooks. I’ve always been disciplined in using formatters and linters in projects, but never enjoyed having them set up to run automatically. This changed when I started working with agents.

I treat pre‑commit as a contract between me and the agent. The agent is instructed to always fix linting/testing failures before returning to me for input or declaring success. In particular, I’ve been using prek, which is a Rust rewrite of pre-commit.

Stricter lints for agents #

With agents, I’m also more inclined toward stricter, more pedantic lints to enforce properties like maximum line count in a function or file. When I’m writing code myself I make these changes instinctively. It’s obvious when a function becomes cumbersome when you’re editing by hand, but I noticed early on that an agent seems much more comfortable with 100+ line functions than I am.

If you care about such constraints (function length, etc.), encode them as lints rather than hoping the agent internalises your preferences.

As an aside, I’d never used flake-parts before, but its ability to automatically configure pre-commit hooks and formatting tools like treefmt in a Nix dev shell is really slick.

Self-updating instructions #

I hadn’t previously realised how good these tools are at writing their own instructions. I had been confused in the past at the length/complexity of AGENTS.md/CLAUDE.md files I’d seen, not realising that when you work with an agent to solve a problem, or remedy something it did, you can then prompt it to summarise what just happened in the CLAUDE.md file to prevent it from happening again.

Plan Mode #

To begin with I made extensive use of “Plan Mode”. The premise is that the agent can explore the codebase, read docs online and work with you to plan new features, but it cannot make changes to the code or your system.

This had been working well for me, but there were a couple of occasions where it seemed to get stuck in a “doom loop” trying to plan its way out of a complex failure.

On one particular occasion, this lasted for over an hour. I then switched to “Agent Mode”, gave it a follow-up prompt that instructed it to inspect a running application on Kubernetes directly using kubectl, and it was able to diagnose and solve the same problem in around 5 minutes. The problem here was that some “tools” are explicitly not permitted in “Plan Mode”, but depending on what you’re trying to achieve that can sometimes be a significant hindrance.

I’ve been favouring a different approach where I remain in “Agent Mode”, and prompt the agent that I’d like to work on a plan in a shared document (e.g. ./plans/new-feature.md), and that it may only edit that file until I say otherwise. I then work in a loop with the Agent where it writes a plan, I leave comments in-line in the file, and then it takes that into account - and repeat until I’m happy. Before asking the agent to implement the plan, I start a new session and ask it to implement the plan with fresh context. So far I’ve found this to be really effective.

Worktrees and Parallel Agents #

Pretty soon after starting work on more complicated features, I became interested in running parallel agents. Sometimes an agent can take a long time to work through a well-developed, and in that time I often felt I could be working my way through other items on my roadmap.

I initially tried to just run them concurrently on the same branch, but predictably they ended up fighting and making a bit of a mess. I’ve long been a fan of git worktrees, and they’re a perfect fit here - enabling me to run concurrent agents on separate instances of the codebase.

Even with worktrees, I still tend to work on orthogonal concerns so I don’t end up with big merge conflicts, but being able to work on a frontend feature, a backend feature and the CI pipeline all at once is a nice upgrade.

Visual prompting #

When working on frontend components, I built a nice workflow around screenshotting the UI, marking it up with Flameshot, then pasting the image back into Claude Code with the next prompt.

I found this to be a really effective way to drive iteration on web components. It felt closer to pair‑designing with a human: I’d circle spacing issues or misaligned elements, and the agent would propose CSS tweaks in response to what it saw.

Summary #

I had a blast building Brewlog, and I’ve been using it every day for about a month now. It’s different from how I’ve approached projects in the past, in a way that’s both liberating and terrifying in equal measures.

While some of the code in this project lacks the same attention to detail I might have applied, I likely would never have got around to building it without an agent.

I was surprised what the process taught me. Every day I hacked on Brewlog with Claude Code, I learned something. Whether that was something about Tailwind CSS, or an architectural pattern.

I think the key to useful output from these tools is that you remain the driver. You’re still responsible what you ship, which means giving the generated code the appropriate amount of review.

Brewlog is live at coffee.jnsgr.uk and runs comfortably on a free-tier instance on fly.io.

If you’re into speciality coffee and like owning your data, give it a try and let me know how you get on.

Finally, I’d like to thank the authors of Roastguide for the amazing work they’ve done on their app, the attention they pay to their community on Discord, and the fact that they were willing to get me a dump of my data I could use to populate the first deployment of Brewlog!

An update on upki

Mon, 16 Feb 2026 00:00:00 +0000

This article was originally posted on the Ubuntu Discourse, and is reposted here. I welcome comments and further discussion in that thread.

Last year, I announced that Canonical had begun supporting the development of upki, a project that will bring browser-grade Public Key Infrastructure (PKI) to Linux. Since then, development has been moving at pace thanks to the tireless work of Dirkjan and Joe.

In this post, I’ll explore the progress we’ve made, how you can try an early version, and where we’re going next.

Architecture & Progress #

As a reminder, upki’s primary goal is to provide a reliable, privacy-preserving, and efficient certificate revocation mechanism for Linux system utilities, package managers, and language runtimes. The solution is built around CRLite, an efficient data format that compresses and distributes certificate revocation information at scale.

The upki repository is structured as a Cargo workspace containing five crates, each serving a distinct role:

upki: the core library and CLI tool. This crate contains the revocation query engine, the client-side sync logic for fetching filter updates, and the command-line interface. The revocation interface was originally embedded in the CLI, but has since been promoted into the library so that other Rust projects can use it directly as a dependency.
upki-mirror: the server-side mirroring tool. This binary fetches and validates CRLite filters from Mozilla’s infrastructure such that they can be served using a standard web server like nginx or apache.
upki-ffi: the C Foreign Function Interface. Built as a cdylib, this crate uses cbindgen to auto-generate a upki.h header file, exposing the revocation query API to C, C++, Go and any other language with C FFI support.
rustls-upki: an integration crate that wires upki’s revocation engine into rustls, enabling any Rust application using rustls to perform CRLite-backed revocation checks transparently.
revoke-test: testing infrastructure for validating revocation queries against known-revoked certificates.

The team recently released v0.1.0, which should help us to gather more feedback on the work we’ve done so far.

How to try it #

If you’d like to try the code in its current form, you’ll need to have a version of the Rust toolchain installed. The easiest way to do this on Ubuntu is using the rustup snap:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# Ensure you have a C compiler in your PATH
sudo apt update
sudo apt install -y build-essential curl

# Install the rustup snap and get the stable toolchain
sudo snap install --classic rustup
rustup install stable

# Install upki
cargo install upki
export PATH="$HOME/.cargo/bin:$PATH"

# Fetch revocation data. This will be done in the background
# when installed through the distro in the future
upki fetch

That should be all you need to install the development version of upki, and you can now use it to run a revocation check by piping certificate output from curl into upki:

1
2


curl -sw '%{certs}' https://google.com | upki revocation check
NotRevoked

Early versions of docs for the C FFI crate and Rust crate documentation are available, but if you’d like to explore, build the project from source, or contribute, the repository is the best place to start. For an example of the C FFI interface in action you can take a look at the upki-go-demo Dirkjan published.

Next Steps #

Now the foundational pieces are in place, our focus is shifting to external consumption, performance, and integration with the wider Linux ecosystem. In the coming days there should be an early 0.1.0 binary release.

We’ll also be doing some performance benchmarking on the initial fetch and of the revocation checks themselves. Currently, each revocation check reads several CRLite filter files into memory. There may be quick wins to improve this, but we’ll benchmark first and see if it warrants optimisation at this time.

We also need to deploy some production infrastructure for serving the CRLite filters. If you follow the steps above, you’ll be fetching from a pre-production web server hosted at https://upki.rustls.dev. We’ve built a Juju charm for operating the CRLite mirror on Kubernetes. This charm packages the upki-mirror binary in a chiselled Rock, and will be deployed into Canonical’s datacentres to serve CRLite data at crlite.ubuntu.com.

Our Ubuntu Foundations team is also working on packaging the various upki components for inclusion in the Ubuntu archive, which will enable you to apt install upki in the future, and also enable us to package and enable it by default in Ubuntu 26.10 and beyond.

Further Down the Road #

While the work above covers what’s immediately in front of us, there is scope to expand upki’s capabilities further. Two areas of interest are Certificate Transparency enforcement, and support for Merkle Tree Certificates.

Certificate Transparency Enforcement #

While upki’s initial focus is on revocation checking, the project also aims to eventually support Certificate Transparency (CT) enforcement. CT is a more modern security measure that relies upon a set of publicly auditable, append-only logs that record every TLS certificate issued by a Certificate Authority (CA). This prevents CAs from issuing fraudulent or erroneous certificates without a means for that fraudulent activity to be discovered - a problem that has bitten organisations in the past.

CT Enforcement would enable clients to refuse to establish a connection unless the server provides cryptographic proof that its certificate has been correctly logged. Browsers like Chrome and Firefox already enforce this, but the rest of the Linux ecosystem would need a tool such as upki to enable such functionality.

Intermediate Preloading #

A correctly configured TLS server should not only send its own certificate, but also the intermediate certificates needed to chain back to a trusted root. In practice, many servers omit the intermediate certificates, and because browsers have quietly worked around this for years, the misconfiguration often goes unnoticed.

Firefox has been preloading all intermediates disclosed to the Common CA Database (CCADB) since Firefox 75, while Chrome and Edge will silently fetch missing intermediates using the Authority Information Access (AIA) extension in the server’s certificate. The result is that a broken certificate chain that works perfectly in every browser will produce an opaque UNKNOWN_ISSUER error when accessed by Linux utilities like curl.

Because upki already maintains a regularly synced local data store, it’s well positioned to ship the known set of intermediates alongside the CRLite filters. This wouldn’t provide a security improvement so much as a usability improvement. It would also bring non-browser clients up to parity with browsers with respect to connection reliability. There is an additional privacy benefit too: rather than fetching a missing intermediate from the issuing CA (which discloses browsing activity to the CA), the intermediate is already present locally.

Merkle Tree Certificates #

Looking even further ahead, upki could support the next generation of web PKI by including support for Merkle Tree Certificates (MTCs). This is an area of active development in the IETF, with Cloudflare and Chrome recently announcing an experimental deployment.

The motivation for MTCs comes largely from the transition to Post-Quantum (PQ) cryptography. PQ signatures are significantly larger than their non-PQ counterparts. The signatures for ML-DSA-44 are 2,420 bytes compared to 64 bytes for ECDSA-P256. A typical TLS handshake today involves multiple signatures and public keys across the certificate chain and CT proofs, which means a simple swap to PQ algorithms would add tens of kilobytes of overhead per connection and likely a noticeable increase in connection latency.

MTCs address this by rethinking how certificates are validated. Rather than transmitting a full certificate chain with multiple signatures, a Certificate Authority can batch certificates into a Merkle Tree and sign only the tree’s root hash. The client then receives just a single signature, a public key, and a compact Merkle tree inclusion proof that demonstrates the certificate’s presence in the batch. The signed tree heads can be distributed to clients out-of-band, meaning the per-handshake overhead is drastically reduced.

Because upki already maintains a local data store that is regularly synced, it could cache tree head data alongside CRLite filters, thereby enabling the inclusion proofs sent during TLS handshakes to be even smaller. Rather than proving inclusion all the way from the leaf to the root, the server could send a “truncated” proof that starts partway up the tree, with the client computing the remainder from data it already has locally. There is a TLS extension being developed to negotiate this.

The implementation of MTCs for TLS is still highly experimental. MTCs are not yet deployed in any browser, but upki will lay the groundwork for Linux system utilities to benefit from this evolution as the technology is adopted.

Summary #

In the few weeks since we announced upki, the core revocation engine has been established and is now functional, the CRLite mirroring tool is working and a production deployment in Canonical’s datacentres is ongoing. We’re now preparing for an alpha release and remain on track for an opt-in preview for Ubuntu 26.04 LTS.

Beyond revocation, we’re keeping a close eye on the evolving PKI landscape and particularly CT enforcement and Merkle Tree Certificates.

I’d like to extend my thanks again to Dirkjan and Joe for their continued collaboration on this work, and the utmost professionalism they’ve demonstrated throughout.

Developing with AI on Ubuntu

Tue, 20 Jan 2026 00:00:00 +0000

This article was originally posted on the Ubuntu Discourse, and is reposted here. I welcome comments and further discussion in that thread.

AI-assisted tooling is becoming more and more common in the workflows of engineers at all experience levels. As I see it, our challenge is one of consideration, enablement and constraint. We must enable those who opt-in to safely and responsibly harness the power of these tools, while respecting those who do not wish to have their platform defined or overwhelmed by this class of software.

The use of AI is a divisive topic among the tech community. I find myself a little in both camps, somewhere between sceptic and advocate. While I’m quick to acknowledge the negative impacts that the use of LLMs can have on open source projects, I’m also surrounded by examples where it has been used responsibly to great effect.

Examples of this include Filippo’s article debugging low-level cryptography with Claude Code, Mitchell’s article on Vibing a Non-Trivial Ghostty Feature, and David’s article How I Program with Agents. These articles come from engineers with proven expertise in careful, precise software engineering, yet they share an important sentiment: AI-assisted tools can be a remarkable force-multiplier when used in conjunction with their lived experience, but care must still be taken to avoid poor outcomes.

The aim of this post is not to convince you to use AI in your work, but rather to introduce the elements of Ubuntu that make it a first-class platform for safe, efficient experimentation and development. My goals for AI and Ubuntu are currently focused on enabling those who want to develop responsibly with AI tools, without negatively impacting the experience of those who’d prefer not to opt-in.

Hardware & Drivers #

AI-specific silicon is moving just as fast as AI software tooling, and without constant work to integrate drivers and userspace tools into Ubuntu, it would be impossible to efficiently utilise this specialised hardware.

Last year we announced that we will ship both NVIDIA’s CUDA and AMD’s ROCm in the Ubuntu archive for Ubuntu 26.04 LTS, in addition to our previous work on OpenVINO. This will make installing the latest drivers and toolkits easier and more secure, with no third-party software repositories. Distributing this software as part of Ubuntu enables us to be proactive in the delivery of security updates and the demonstration of provenance.

Our work is not limited to AMD and NVIDIA; we recently announced support for Qualcomm’s Dragonwing platforms and others. You can read more about our silicon partner projects on our website.

Inference Snaps #

At the Ubuntu Summit 25.10, we released “Inference Snaps” into the wild, which provide a hassle-free mechanism for obtaining the “famous model” you want to work with, but automatically receive a version of that model which is optimised for the silicon in your machine. This removes the need to spend hours on HuggingFace identifying the correct model to download that matches with your hardware, and obviates the need for in-depth understanding of model quantisation and tuning when getting started.

Each of our inference snaps provide a consistent experience: you need only learn the basics once, but can apply those skills to different models as they emerge, whether you’re on a laptop or a server.

At the time of writing, we’ve published beta quality snaps for qwen-vl, deepseek-r1 and gemma3. You can find a current list of snaps in the documentation, along with the silicon-optimised variants.

Sandboxing Agents #

While many start their journey in a web browser chatting to ChatGPT, Claude, Gemini, Perplexity or one of the myriad of alternatives, many developers will find “agentic” tools such as Copilot, Codex, Claude Code or Amp quite attractive. In my experience, agents are a clear level-up in an LLM’s capability for developers, but they can still make poor decisions and are generally safer to run in sandboxed environment at the time of writing.

Where a traditional chat-based AI tool responds reactively to user prompts within a single conversation, an agent operates (semi-)autonomously to pursue goals. It perceives its environment, plans, makes decisions and can call out to external tools and services to achieve those goals. If you grant permission, an agent can read and understand your code, implement features, troubleshoot bugs, optimise performance and many other tasks. The catch is that they often need access to your system - whether that be to modify files or run commands.

Issues such as accidental file deletion, or the inclusion of a spurious (and potentially compromised) dependency are an inevitable failure mode of the current generation of agents due to how they’re trained (see the Reddit post about Claude Code deleting a user’s home directory).

My agent sandboxes itself! #

Some of you will be reading this wondering why additional sandboxing is required, since many of the popular agents advertise their own sandboxing. The fact that some agents include some measures to protect the user’s machine is of course a good thing. The touted benefits include filesystem isolation by restricting the agent to a specific directory, or prompting for approval before modifying files. Some agents also include network sandboxing to restrict network access to a list of approved domains, or by using a custom proxy to impose rules on outbound traffic.

On Linux, these agent-imposed sandboxes are often implemented with bubblewrap, which is “a tool for constructing sandbox environments”, but note that the upstream project’s README includes a section which states that it is not a “ready-made sandbox with a specific security policy”. bubblewrap is a relatively low-level tool that must be given its configuration, which in this case is provided by the agent.

The limitation upon these tools is the shared kernel - a severe kernel exploit could enable an agent to escape from its sandbox. Of course, such vulnerabilities are rare, but note that even if the sandboxing technologies do their job, agents often run in the context of the user’s session, meaning they inherit environment variables which could contain sensitive information. They’re also agent specific: Claude Code’s sandboxing won’t help you if you’re using Cursor or Antigravity.

Depending on your threat model and the project you’re working on, you may deem the built-in sandboxing of coding agents to be sufficient, but there are other options available to Ubuntu users that provide either different, or additional protection…

Sandbox with LXD containers #

Canonical’s LXD works out-of-the-box on Ubuntu, and is a great way to sandbox an agent into a disposable environment where the blast radius is limited should the agent make a mistake. My personal workflow is to create an Ubuntu container (or VM) with my project directory mounted. This way, I can edit my code directly on my filesystem with my preferred (already configured) editor, but have the agent run inside the container.

For example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Initialise the container
lxc init ubuntu:noble dev
# Mount my project directory into the container
lxc config device add -q dev datadir disk source="$HOME/my-project" path=/home/ubuntu/project
# Start the container
lxc start dev
# Get a shell inside the container as the 'ubuntu' user
lxc exec dev -- sudo -u ubuntu -i bash
# Run a command in the container
lxc exec dev -- sudo -u ubuntu -i bash -c "cd project; claude"

You can learn more about LXD in the official documentation and tutorial, as well as specific instructions on enabling GPU data processing in containers/VMs. I’ve written previously about my use of LXD in development.

With LXD, you can choose between running your sandbox as a container or a VM, depending on your project’s needs. If I’m working on a project that requires Kubernetes or similar, I use a VM, but for lighter projects I use system containers, preferring their lower overhead.

Sandbox with LXD VMs #

LXD is best known for its ability to run “system containers”, which are somewhat analogous to Docker/OCI containers, but rather than being focused on a single application (and dependencies), a system container essentially runs an entire Ubuntu user-space (including systemd, etc.). Like OCI containers, however, system containers share the kernel with the host.

In some situations, you may seek more isolation from your host machine by running tools inside a virtual machine with their own kernel. LXD makes this simple - you can follow the same commands as above, but add --vm to the init command:

1
2


# Initialise the virtual machine
lxc init --vm ubuntu:noble dev

You can also configure the virtual machine’s CPU, memory and disk requirements. A simple example is below:

1
2
3
4


lxc init --vm ubuntu:noble dev \
 -c limits.cpu=8 \
 -c limits.memory=8GiB \
 -d root,size=100GiB

You can find more details on instance configuration in the LXD documentation.

Sandbox with Multipass #

Multipass provides on-demand access to Ubuntu VMs from any workstation - whether that workstation is running Linux, macOS or Windows. It is designed to replicate, in a lightweight way, the experience of provisioning a simple Ubuntu VM on a cloud.

Multipass’ scope is more limited than LXD, but for many users it provides a simple on-ramp for development with Ubuntu. Where it lacks advanced features like GPU passthrough, it boasts a simplified CLI and a first-class GUI client.

To get started similarly to the LXD example above, try the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Install Multipass
sudo snap install multipass
# Launch an instance
multipass launch noble -n dev
# Mount your project directory
multipass mount ~/my-project dev:/home/ubuntu/project
# Get a shell in the instance
multipass shell dev
# Run a command in the instance
multipass exec dev -- claude

You can find more details on how to configure and manage instances in the docs.

Sandbox with WSL #

If you’re on Windows, development with WSL includes first-class support for GPU acceleration, and is even supported for use with the NVIDIA AI Workbench, NVIDIA NIM and CUDA.

Ubuntu is the default Linux distribution for WSL, and you can find more information about how to set up and use Ubuntu on WSL in our documentation. WSL benefits from all the same technologies as a “regular” Ubuntu install, including the ability to use Snaps, Docker and LXD.

For the enterprise developer, we recently announced Ubuntu Pro for WSL, as well as the ability to manage WSL instances using Landscape, making it easier to get access to first-class developer tooling with Ubuntu on your corporate machine.

Summary #

While opinion remains divided on the value and impact of current AI tooling, its presence in modern development workflows and its demands on underlying compute infrastructure are difficult to ignore.

Developers who wish to experiment need reliable access to modern hardware, predictable tooling, and strong isolation boundaries. Ubuntu’s role is not to dictate how these tools are used, but to provide a stable and dependable platform on which they can be explored and deployed safely, without compromising security, provenance, or the day-to-day experience of those who choose to opt out.

In addition to powering development workflows, Ubuntu makes for a dependable production operating system for your workloads. We’re building Canonical Kubernetes with first-class GPU support, Kubeflow and MLFlow for model training and serving and a suite of applications like PostgreSQL, MySQL, Opensearch, as well as other data-centric tools such as Kafka and Spark that can be deployed with full Ubuntu Pro support. Let me know if you’d find value in a follow-up post on those topics!

Addressing Linux's Missing PKI Infrastructure

Mon, 08 Dec 2025 00:00:00 +0000

This article was originally posted on the Ubuntu Discourse, and is reposted here. I welcome comments and further discussion in that thread.

Earlier this year, LWN featured an excellent article titled “Linux’s missing CRL infrastructure”. The article highlighted a number of key issues surrounding traditional Public Key Infrastructure (PKI), but critically noted how even the available measures are effectively ignored by the majority of system-level software on Linux.

One of the motivators for the discussion is that the Online Certificate Status Protocol (OCSP) will cease to be supported by Let’s Encrypt. The remaining alternative is to use Certificate Revocation Lists (CRLs), yet there is little or no support for managing (or even querying) these lists in most Linux system utilities.

To solve this, I’m happy to share that in partnership with rustls maintainers Dirkjan Ochtman and Joe Birr-Pixton, we’re starting the development of upki: a universal PKI tool. This project initially aims to close the revocation gap through the combination of a new system utility and eventual library support for common TLS/SSL libraries such as OpenSSL, GnuTLS and rustls.

The Problem #

Online Certificate Authorities responsible for issuing TLS certificates have long had mechanisms for revoking known bad certificates. What constitutes a known bad certificate varies, but generally it means a certificate was issued either in error, or by a malicious actor of some form. There have been two primary mechanisms for this revocation: Certificate Revocation Lists (CRLs) and the Online Certificate Status Protocol (OCSP).

In July 2024, Let’s Encrypt announced the deprecation of support for the Online Certificate Status Protocol (OCSP). This wasn’t entirely unexpected - the protocol has suffered from privacy defects which leak the browsing habits of users to Certificate Authorities. Various implementations have also suffered reliability issues that forced most implementers to adopt “soft-fail” policies, rendering the checks largely ineffective.

The deprecation of OCSP leaves us with CRLs. Both Windows and macOS rely on operating system components to centralise the fetching and parsing of CRLs, but Linux has traditionally delegated this responsibility to individual applications. This is done most effectively in browsers such as Mozilla Firefox, Google Chrome and Chromium, but this has been achieved with bespoke infrastructure.

However, Linux itself has fallen short by not providing consistent revocation checking infrastructure for the rest of userspace - tools such as curl, system package managers and language runtimes lack a unified mechanism to process this data.

The ideal solution to this problem, which is slowly becoming more prevalent, is to issue short-lived credentials with an expiration of 10 days or less, somewhat removing the need for complicated revocation infrastructure, but reducing certificate lifetimes is happening slowly and requires significant automation.

CRLite #

There are several key challenges with CRLs in practice - the size of the list has grown dramatically as the web has scaled, and one must collate CRLs from all relevant certificate authorities in order to be useful. CRLite was originally proposed by researchers at IEEE S&P and subsequently adopted in Mozilla Firefox. It offers a pragmatic solution to the problem of distributing large CRL datasets to client machines.

In a recent blog post, Mozilla outlined how their CRLite implementation meant that on average users “downloaded 300kB of revocation data per day, a 4MB snapshot every 45 days and a sequence of “delta-updates” in-between”, which amounts to CRLite being 1000x more bandwidth-efficient than daily CRL downloads.

At its core, CRLite is a data structure compressing the full set of web-PKI revocations into a compact, efficiently queryable form. You can find more information about CRLite’s design and implementation on Mozilla’s Security Blog.

Introducing upki #

Following our work on oxidizing Ubuntu, Dirkjan reached out to me with a proposal to introduce a system-level utility backed by CRLite to non-browser users.

upki will be an open source project, initially packaged for Ubuntu but available to all Linux distributions, and likely portable to other Unix-like operating systems. Written in Rust, upki supports three roles:

Server-side mirroring tool: responsible for downloading and mirroring the CRLite filters provided by Mozilla, enabling us to operate independent CDN infrastructure for CRLite users, and serving them to clients. This will insulate upki from changes in the Mozilla backend, and enable standing up an independent data source if required. The server-side tool will manifest as a service that periodically checks the Mozilla Firefox CRLite filters, downloads and validates the files, and serves them.
Client-side sync tool: run regularly by a systemd-timer, network-up events or similar, this tool ensures the contents of the CDN are reflected in the on-disk filter cache. This will be extremely low on bandwidth and CPU usage assuming everything is up to date.
Client-side query tool: a CLI interface for querying revocation data. This will be useful for monitoring and deployment workflows, as well as for users without a good C FFI.

The latter two roles are served by a single Rust binary that runs in different modes depending on how it is invoked. The server-side tool will be a separate binary, since its use will be much less widespread. Under the hood, all of this will be powered by Rust library crates that can be integrated in other projects via crates.io.

For the initial release, Canonical will stand up the backend infrastructure required to mirror and serve the CRLite data for upki users, though the backend will be configurable. This prevents unbounded load on Mozilla’s infrastructure and ensures long-term stability even if Firefox’s internal formats evolve.

Ecosystem Compatibility #

So far we’ve covered the introduction of a new Rust binary (and crate) for supporting the fetching, serving and querying of CRL data, but that doesn’t provide much service to the existing ecosystem of Linux applications and libraries in the problem statement.

The upki project will also provide a shared object library for a stable ABI that allows C and C-FFI programs to make revocation queries, using the contents of the on-disk filter cache.

Once upki is released and available, work can begin on integrating existing crypto libraries such as OpenSSL, GNUtls and rustls. This will be performed through the shared object library by means of an optional callback mechanism these libraries can use to check the revocation lists before establishing a connection to a given server with a certificate.

Timeline #

While we’ve been discussing this project for a couple of months, ironing out the details of funding and design, work will soon begin on the initial implementation of upki.

Our aim is to make upki available as an opt-in preview for the release of Ubuntu 26.04 LTS, meaning we’ll need to complete the implementation of the server/client functionality, and bootstrap the mirroring/serving infrastructure at Canonical before April 2026.

In the following Ubuntu release cycle, the run up to Ubuntu 26.10, we’ll aim to ship the tool by default on Ubuntu systems, and begin work on integration with the likes of NSS, OpenSSL, GNUtls and rustls.

Summary #

Linux has a clear gap in its handling of revocation data for PKIs. Over the coming months we’re hoping to address that gap by developing upki not just for Ubuntu, but for the entire ecosystem. Thanks to Mozilla’s work on CRLite, and the expertise of Dirkjan and Joe, we’re confident that we’ll deliver a resilient and efficient solution that should make a meaningful contribution to systems security across the web.

If you’d like to do more reading on the subject, I’d recommend the following:

LWN.net: Linux’s missing CRL infrastructure
Mozilla Security Blog: CRLite Part 1: All Web PKI Revocations Compressed
Mozilla Security Blog: CRLite Part 2: End-to-End Design
Let’s Encrypt: Replacing OCSP with CRLs
IEEE Symposium on Security & Privacy: CRLite: A Scalable System for Pushing All TLS Revocations to All Browsers

Ubuntu Summit 25.10: Personal Highlights

Sun, 02 Nov 2025 00:00:00 +0000

This article was originally posted on the Ubuntu Discourse, and is reposted here. I welcome comments and further discussion in that thread.

I recently had the privilege of attending the Ubuntu Summit 25.10. Ubuntu Summits have a relatively long history. Some years ago Canonical ran the ‘Ubuntu Developer Summits (UDS)’, but recently the events were brought back and reimagined as the ‘Ubuntu Summit’.

For the most recent Summit, we tried out a new format. We invited a select few folks to come and give talks at our London office, with a small in-person crowd. In addition, the event was livestreamed, and we encouraged people to host “watching parties” across the world as part of Ubuntu Local Communities (LoCos).

While Ubuntu may feature in the name, the event does not require talks to be centred on Ubuntu, and in fact is aiming to draw contributions from our partners and from right across the open source community, whether or not the content is relevant to Ubuntu or Canonical - it’s designed to be a showcase for the very best of open source, and this year I felt that the talks were of a particularly high calibre.

In this post I’ll highlight some of my favourite talks, in no particular order! If any of these catch your interest, you can see when they were aired and catch-up on the Day 1 and Day 2 streams.

DOOM in Space #

What a way to kick off the Summit! DOOM in Space was a talk given by Ólafur Waage, who introduced himself as a “professional keyboard typist”!

The talk was immediately after Mark Shuttleworth’s opening remarks, and covered his journey in getting DOOM to run on the European Space Agency’s OPS-SAT satellite. DOOM has famously been ported to many devices, though some were only questionably “running” the game.

Ólafur covered how he became involved in the project, and the unique approach they needed to take to guarantee success, since they would only get a very limited amount of time in order to conduct their “experiment” on the satellite.

Of particular note was the work done to integrate imagery from the OPS-SAT’s onboard camera into the game, which involved some clever reassigning of colors in the game’s original palette to more faithfully represent the imagery taken from the camera in-game.

Infrastructure-Wide Profiling of Nvidia CUDA #

This talk was given by Frederic Branczyk, CEO and Founder of Polar Signals. Canonical has partnered with Polar Signals a couple of times in recent years. They were part of our journey to enabling frame pointers by default on Ubuntu, and many of our teams have been using their zero-instrumentation eBPF profiler.

While CPU profiling has been commonplace for developers for many years, giving the ability to analyse CPU and memory-bound workloads, profiling GPU workloads has been less prominent, and is particularly difficult in production.

Polar Signals advocate for “continuous profiling”, which means running a profiler at all times, on all nodes, in production. The benefit of this is that when an issue occurs, you don’t have to set up a profiler and try to reproduce the issue - you already have the data. It also negates the uncertainty of the impact a profiler might have on the code during reproduction. This would have been difficult with traditional profiling tools, but with technologies like eBPF, the overhead of the profiler is incredibly low compared to the potential performance gains from acting on the data it produces.

In this talk, Frederic outlined the work they have done bringing infrastructure-wide profiling of CUDA workloads into Polar Signals Cloud. Their approach combines the CUPTI profiling API with USDT probes and eBPF into a pipeline, relying upon the ability to inject a small library into CUDA workloads using the CUDA_INJECTION64_PATH without modification.

You can see more details on their website.

Inference Snaps #

This talk served as the first public announcement of Inference Snaps from Canonical, which represents a few months of working combining many of the new technologies behind Snaps.

As Large Language Models continue to gain pace along with the rest of the AI community, silicon manufacturers are increasingly including dedicated hardware in commodity CPUs and GPUs, as well as shipping dedicated accelerators for some workloads.

AI models often need to be tuned in some way in order to work optimally - for example quantisation which aims to reduce the computational memory costs of running inference on a given model.

Inference snaps provide a hassle-free mechanism for users to obtain the “famous model” they want to work with, but automatically receive a version of that model which is optimised for the silicon in their machine, removing the need to spend hours on HuggingFace trying to identify the correct model to download that matches with their hardware.

Using our extensive partner network, we’ll continue to work with multiple silicon vendors to ensure that models are available for the latest hardware as it drops, and provide a consistent experience to Ubuntu users that wish to work with AI.

Find out more in the announcement.

Nøughty Linux: Ubuntu’s Stability Meets Nixpkgs’ Freshness #

This talk was a bit of a guilty pleasure for me! Delivered by Martin Wimpress (wimpy), the audience were shown how they could take a stock Ubuntu Server deployment, and use a collection of scripts to layer a cutting-edge GUI stack on top using Nixpkgs.

Wimpy outlined his motivation as wanting to rely upon the stable kernel and hardware support offered by Ubuntu, but wanting to be more experimental with his desktop environment and utilities - preferring a tiling window management experience.

Having spent some years on NixOS, Wimpy was recently required to run a security “agent” for work, which was very difficult to enable on NixOS, but worked out of the box on Ubuntu. Recognising the need to make the switch, he was reluctant to move away from the workflow he’d built so much muscle-memory around - and so Nøughty Linux was born!

Nøughty Linux is not a Linux distribution, rather a set of configurations for an Ubuntu Server machine. It utilises nix-system-graphics and system-manager and is actually very similar to a configuration I ran in my own nixos-config repository for my laptop for a while - though Wimpy has chased down significantly more of the papercuts than I did!

Are we stuck with the same Desktop UX forever? #

Scott Jenson delivered an incredibly engaging talk in which he posited that desktop user experience has somewhat stagnated, and worse that many of the patterns we’ve become used to on the desktop are antiquated and unergonomic.

The crux of the talk was to focus on user experience, rather than user interfaces - challenging developers to think about how people learn, and how desktops could benefit more from design affordances by rethinking some critical elements such as window management or text editing.

Using his years of experience at Apple, Symbian and Google, Scott delivered one of the most engaging conference talks I’ve seen, and I thoroughly recommend watching it on our YouTube channel!

Honorable Mentions #

In addition to the talks above, it was a delight to meet Mark Schoolderman from the Trifecta Tech Foundation in-person, who led the work on sudo-rs as part of our “Oxidising Ubuntu” story, and interesting to hear about the value the project derived from Ubuntu’s Main Inclusion Review process as part of landing sudo-rs in main.

Equally, I was delighted that Samuele Kaplun from Proton could join us to talk about the work we’ve been doing together on bringing first-class Snap packages for Proton Mail, Proton VPN, Proton Pass and Proton Authenticator to the Snap store, and their reasons for choosing Snaps, adventures with confinement, and more.

I was delighted to see Craig Loewen and Clint Rutkas present on their journey open sourcing the Windows Subsystem For Linux (WSL), which represents a growing proportion of Ubuntu users, and a key bridge to open source development for many.

Finally, thank you to Utkarsh for this wonderful slide as part of his talk on Ubuntu Snapshot Releases:

Conclusion #

Overall, I found the Ubuntu Summit 25.10 a really enjoyable event, with talks that were uniformly high in quality, charisma and creativity. I’m pleased that Canonical has broadened the Summit’s reach and I hope it continues to serve as a platform to showcase the very best open source innovation.

Until next time!

Ubuntu Engineering in 2025: A Retrospective

Thu, 09 Oct 2025 00:00:00 +0000

This article was originally posted on the Ubuntu Discourse, and is reposted here. I welcome comments and further discussion in that thread.

Ubuntu 25.10: A Retrospective #

In February this year, I published Engineering Ubuntu For The Next 20 Years, which was something of a manifesto I pledged to enact in the design, build and release of Ubuntu. This week, we released Ubuntu 25.10 Questing Quokka, which was the first full engineering cycle under this new manifesto, and it seems like a good time to reflect on what we achieved in each category, as well as highlight some of the more impactful changes that have just landed in Ubuntu.

In that first article, I outline four themes for Ubuntu Engineering at Canonical to focus on: Communication, Automation, Process and Modernisation.

Communication #

A notable improvement throughout this engineering cycle has been the frequency with which the teams at Canonical have written about their work, often in some detail. Many of these posts can be found under the blog tag, which had never been used until around six months ago, and now sees a couple of new posts per week outlining the work people are doing toward these themes.

I stated that I consider documentation a key part of our communication strategy, and this last six months has seen some of the most substantial changes to Ubuntu documentation in many years. The Ubuntu Project Docs was a project started in May 2025, and is quickly becoming the single documentation hub that a current or potential Ubuntu contributor needs to understand how, why and when to do their job. Similarly, the Ubuntu for Developers was created to illuminate a path for developers across numerous languages on Ubuntu.

It’s important for us to celebrate such efforts, but also to remember that this is only the start! In order for these efforts to remain useful, both our internal teams and our community must continue to engage with these efforts - adding, refining and pruning content as necessary. As the sun-setting of wiki.ubuntu.com approaches, it’s imperative that these new documentation sites continue to get the attention they need.

Lots of the changes we’ve made in the last cycle have attracted attention from online blogs, news outlets, youtubers, etc. Part of the challenge with such changes is “owning the narrative” and ensuring that legitimate concerns are heard (and taken into account), but also that there are appropriate responses to uncertainty, without getting drawn into unproductive discussions.

Finally, the transition to Matrix as the default synchronous communication means for the project has, in my opinion, made it easier than ever to get in touch with our community of experts - whether it be for support, or to start a journey for contribution to Ubuntu.

Automation #

The largest item we took on here was in pursuit of the monthly snapshot releases. This went much better than I expected, and to some extent covers off the “Process” theme as well as “Automation”, but through a combination of studying our process and whittling it down as lean as we could, and beginning to automate more of the process, the team were able to release four snapshot releases before the 25.10 Beta.

The scale of the automation efforts was relatively limited this cycle, but the automation of release testing has really accelerated in the past few months. The vast majority of the test cases that qualify an Ubuntu Desktop ISO for release are now fully automated, and the same framework that makes this possible was also used to develop a suite of tests for TPM FDE.

Work was also done on our craft tools to better the experience with the test sub-command of build tools like snapcraft, rockcraft and charmcraft - all of which will have a trickle-down effect on the upcoming debcraft, and make it trivial to include many new kinds of tests in our packaging workflows.

Behind the scenes, every team in Ubuntu Engineering at Canonical has been writing charms that make the underlying infrastructure behind Ubuntu more portable, resilient and scalable. This includes services like Ubuntu Manpages, autopkgtest, error-tracker, and a staging deployment of Temporal to enable the next phase of our release automation.

Process #

This item was probably where the least concrete progress was made, though I probably could have predicted that. Many of the processes in the Ubuntu project serve to ensure that we ship resilient software, and don’t break users - so changing them in a hurry is not generally a good idea.

That said, there was some good progress on the Main Inclusion Review (MIR) process, whose team documentation was moved into the Ubuntu Project Docs after a thorough review, and the Stable Release Updates (SRU) team are in the process of the same transition. Moving and re-reviewing the documentation is essentially the first step of the process improvement I was seeking: understanding where we are!

Internally, we’ve been piloting a new process for onboarding Ubuntu Developers that sees engineers start by working toward gaining upload rights for a single package, but has a complete curriculum that can take them through to Core Developer status. Details of this should be released in the coming months, outlining a clear and well-trodden journey for new contributors. Much of this material already existed, but the team have worked on polishing it, and making it clearer how the process work from end to end.

The next step for each of these processes is measurement. We’ve begun instrumenting these processes to understand where the most time is spent so we can use that information to guide improvements and streamline processes in future cycles, and even set Service Level Objectives (SLOs) against those timelines.

Modernisation #

Much of what I’ve already described could be considered modernisation, but from a technical standpoint the most obvious candidate here was the “Oxidising Ubuntu” effort, which has seen us replace numerous core utilities in Ubuntu 25.10 with modern Rust rewrites.

We began this effort in close collaboration with the uutils project and the Trifecta Tech Foundation. The former is the maintainer of a Rust coreutils rewrite, and the latter the maintainer of sudo-rs, which we made the default in 25.10. The technical impact of these changes in defaults will only truly be known once Ubuntu 25.10 is “out there”, but I’m pleased with how we approached the shift. In both cases, we contacted the upstreams in good time to ascertain their view on their projects’ readiness, then agreed funding to ensure they had the financial support they needed to land changes in support of Ubuntu, and then worked closely with them throughout the cycle to solve various performance and implementation issues we discovered along the way.

As it stands today, sudo-rs is the default sudo implementation on Ubuntu 25.10, and uutils’ coreutils has mostly replaced the GNU implementation, with a few exceptions, many of which will be resolved by releases in the coming weeks. These diversions back to the existing implementations demonstrate that stability and resilience are more important than “hype” in our approach: I expect us to have completed the migration during the next cycle, but not before the tools are ready.

Following the “Switch to Dracut” specification, Ubuntu Desktop 25.10 will use Dracut as its default initrd infrastructure (replacing initramfs-tools). Dracut will use systemd in the initrd and supports new features like Bluetooth and NVMe over Fabric (NVM-oF) support. Ubuntu Server installations will continue using initramfs-tools until remaining hooks are ported.

For each of these changes (coreutils, sudo-rs and dracut) the previous implementations will remain supported for now, with well-documented instructions on the reversion of each change for those who run into unavoidable issues - though we expect this to be a very small number of cases.

What’s Next? #

Well… more of the same! We intend to carry on with the increased cadence of written updates, so keep an eye out for those.

We have some exciting announcements to make over the coming weeks, including support for more modern micro-architectural variants (like amd64v3), better system-wide handling of revoked TLS certificates, updates on our Debcraft package for a more modern packaging experience and an effort to update many of our tools from “behind the scenes” using a combination of Rust and Go.

My final words are to thank all of those who have driven these efforts. I’ll omit the long list of names, but there have been countless examples of people stepping up substantially to deliver these efforts - without whom we’d have made a lot less progress.

Well done, and let’s make Resolute Raccoon an LTS to remember - for all the right reasons!

The Immutable Linux Paradox

Mon, 01 Sep 2025 00:00:00 +0000

This article was originally posted on the Ubuntu Discourse, and is reposted here. I welcome comments and further discussion in that thread.

Immutable Linux distributions have been around since the early 2000s, but adoption has significantly accelerated in the last five years. Mainstream operating systems (OSes) such as macOS, Android, ChromeOS and iOS have all embraced similar principles, reflecting a growing trend toward resilience, longevity, and maintainability as core ideals of OS development.

Ubuntu Core has been at the forefront of this movement for IoT, appliances and edge deployments, with work ongoing to release a “Core Desktop” experience. Other projects such as NixOS, Fedora Silverblue and Red Hat image mode are gaining adoption, alongside more specialised immutable distributions such as SteamOS and Talos.

This post explores how different Linux distributions achieve immutability, the trade-offs, and why you should give it a try!

What is an immutable Linux distribution? #

The key principle of an immutable OS is that the core system is unchangeable at runtime.

Every OS installation has at least one filesystem that stores system software, user software, and user data. Immutable OSes must cleanly separate “system” and “user” software and data, such that regular user interactions cannot compromise the integrity of the OS.

Immutable deployments are often separated into three layers:

Base OS - immutable core, updated only through controlled mechanisms
Applications - user applications, often delivered in containerised formats such as Snap, Flatpak, AppImage, cpak
User data - writable and persistent, independent of OS updates or rollbacks

Immutable systems use atomic, transactional updates meaning updates are applied as unitary, indivisible operations that either wholly succeed, or fail completely and trigger an automated roll-back to a previous known-good state.

Why immutability? #

The major benefit of an immutable OS is resilience.

Immutable OSes make it easier to reproduce systems with a given configuration, which is particularly useful in scale-out use-cases such as cloud or IoT.

Traditional package managers often maintain a database of installed packages, consisting of those included in the base OS, and those explicitly installed by the user, and their dependencies. The package manager doesn’t have a clear notion of which packages make up the “core system”, and which are “optional”.

This can cause “configuration drift”, which occurs over time - a package could be explicitly installed by a user, used for a while and then removed, but without removing its dependencies. This leaves the system in a different, and somewhat undefined, state than it was in prior to the package being installed.

Often the traditional notion of OS security is improved with immutable OS concepts too. In most implementations, the core OS files are mounted read-only such that users cannot make changes - which also raises the bar for malicious modifications. When combined with technologies such as secure boot and confinement, immutable OSes can dramatically reduce the attack surface of a machine.

Finally, convenience! Immutable OSes often include recovery or rollback features, which enable users to “undo” a bad system change, reverting to a previous known-good revision.

The immutability paradox #

In reality, no general-purpose operating system is fully immutable.

There is always persistent, user-writable storage - because without this there would be a huge limitation on usefulness! Similarly, how can a system be truly immutable, yet still support software updates?

The terms “immutable” and “stateless” are often conflated - when in reality neither are excellent terms for describing what has become widely known as “immutable OSes”. This was explored in some depth in this blog post which proposes terms such as “image based” and “fully managed”.

By definition, changes to configuration, the installation of applications and the use of temporary runtime storage are all violations of immutability, and thus immutability concepts must be applied in some sort of layering system.

Striking the balance between ’true’ immutability and user experience is one of the hardest challenges in immutable OS design. A system that is too rigid can be difficult to manage and use, appearing inflexible to end users.

A common pattern is to run an immutable desktop OS and use virtualisation or containerisation technologies (e.g. LXD, Podman, toolbx Distrobox) to create mutable environments in which to work on projects. This results in a very stable workstation that benefits from immutability, with the flexibility of a traditional mutable OS where it’s needed.

Approaches to immutability #

Different distributions solve the immutability challenge in different ways. In this section we’ll explore the four different approaches of ostree based distributions, bootc based distributions, NixOS and Ubuntu Core.

Fedora Silverblue / CoreOS / EndlessOS (`ostree`) #

Fedora Silverblue and Fedora CoreOS are also popular choices for those exploring immutable OSes. The two share a lot of underlying technology with Silverblue targeting desktop use cases, and CoreOS targeting server deployments.

Both are based on ostree, which provides:

tools that combine a ‘git-like’ model for committing and downloading bootable filesystem trees, along with a layer for deploying them and managing the bootloader configuration.

Silverblue and CoreOS actually rely on rpm-ostree , a “hybrid image/package manager” which combines RPM packaging technology with ostree to manage deployments.

The update mechanism involves switching the filesystem to track a different remote “ref”, which is analogous to a git ref.

EndlessOS is based on Debian, but uses ostree to achieve immutability. EndlessOS is a desktop experience designed more for the “average user” and focuses on providing a reliable system that works well in low-bandwidth or offline situations.

Users often use Flatpak to install graphical user applications atop the immutable base, or a user-space package manager such as brew for other utilities.

ostree based distributions also support “package layering” which enables adding packages to the base system without fetching a whole new filesystem ref, but does require the system to be rebooted before the package is persistently available. The documentation notes that this approach is to be used “sparingly”, and that users should prefer using Flatpak or toolbx to access additional packages.

RHEL “Image Mode” (`bootc`) #

bootc based distributions use an alternate approach, packaging the base system into OCI containers (commonly referred to as Docker containers). Atomicity and transactionality are achieved by using container images to deliver the entire core system, and rebooting into a new revision.

RHEL Image Mode uses bootc. This technology capitalises on the success of OCI containers as a transport and delivery mechanism for software by packing an entire OS base image into a single container, including the kernel image.

The bootc project builds on ostree , but where ostree never delivered an opinionated “install mechanism”, bootc does. The contents of a bootc image is an ostree filesystem.

Installing new system packages generally means building a new base image, downloading that image and rebooting into it with a command such as bootc switch <image reference>.

Users often use Flatpak to install graphical user applications atop the immutable base, or a user-space package manager such as brew for other utilities.

NixOS #

The Nix project first appeared in 2003. NixOS is built on top of the Nix package manager, using it to manage both packages and system configuration.

NixOS defines the entire system through a declarative configuration, with changes applied via “generations” that can be rolled back. Changes to the system are applied by “rebuilding” the system configuration, which produces a new “generation”.

Nix packages, and therefore NixOS, eschews the traditional Unix FHS in favour of the Nix “store” and a collection of symlinks and wrappers managed by Nix. Only the Nix package manager can write to the store.

The Nix store also (mostly) enables the building and switching of generations without a reboot. Updates are atomic: new generations must build completely before they can be activated. The home-manager project extends these concepts to the user environment and dotfile management.

The impermanence project requires that every persistent directory is explicitly labelled, or else it’s deleted on every reboot, forcing the base OS to be rebuilt from the Nix store and system configuration - essentially “enforcing” core system immutability between reboots. This was inspired by blog posts “Erase Your Darlings” and “NixOS tmpfs as root”, which are worth a read, too!

Ubuntu Core #

Ubuntu Core achieves immutability by packaging every component (kernel, base system, applications) as Snaps.

Snap confinement enforces isolation, and snapd manages transactional updates and rollbacks. The system is designed for reliability, fleet management, and modular upgrades, making it well-suited for IoT and soon, desktop use.

The key components of an Ubuntu Core deployment are:

Gadget snap: provides boot assets, including board specific binaries and data (bootloader, device tree, etc.)
Kernel snap: kernel image and associated modules, along with initial ramdisk for system initialisation
Base snap: execution environment in which applications run - includes “base” Ubuntu LTS packages
System snaps: packages critical to system function such as Network-Manager, bluez, pulseaudio, etc.
Application snaps: define the functionality of the system, confined to a sandbox
Snapd: manages updates, rollbacks and snapshotting/restoring of user data

In a Core Desktop installation, the desktop environment (GNOME, Plasma, etc.), display manager, login manager would all be delivered as “system snaps”.

Snap confinement ensures packages cannot incorrectly interact with the underlying system or user data without explicit approval. In an Ubuntu Core deployment, this notion is extended to every component of the OS, offering a straightforward yet powerful way to manage risk for each system component.

Summary #

Immutable Linux distributions approach the immutability paradox differently. We explored four different approaches here, and you can learn more about other approaches taken by the likes of SUSE MicroOS (filesystem based immutability) and Vanilla OS (uses ABRoot) in this excellent blog post.

Ubuntu Core focuses on transactional packaging and a clean separation of system & user data. bootc-based systems take a full image-based approach, while NixOS offers extreme flexibility through declarative configuration, but at the cost of complexity.

If you’ve yet to try an immutable Linux distribution I’d recommend giving it a go. Whether you prioritise simplicity, security or declarative control there’s almost certainly an immutable Linux distribution that fits your needs.

Crafting Your Software

Mon, 21 Jul 2025 00:00:00 +0000

This article was originally posted on the Ubuntu Discourse, and is reposted here. I welcome comments and further discussion in that thread.

Packaging software is notoriously tricky. Every language, framework, and build system has its quirks, and the variety of artifact types — from Debian packages to OCI images and cloud images — only adds to the complexity.

Over the past decade, Canonical has been refining a family of tools called “crafts” to tame this complexity and make building, testing, and releasing software across ecosystems much simpler.

The journey began on 23rd June 2015 when the first commit was made to Snapcraft, the tool used to build Snap packages. For years, Snapcraft was the only craft in our portfolio, but in the last five years, we’ve generalized much of what we learned about building, testing, and releasing software into a number of “crafts” for building different artifact types.

Last month, I outlined Canonical’s plan to build debcraft as a next-generation way to build Debian packages. In this post I’ll talk about what exactly makes a craft, and why you should bother learning to use them.

Software build lifecycle #

At the heart of all our crafts is craft-parts, which according to the documentation “provides a mechanism to obtain data from different sources, process it in various ways, and prepare a filesystem sub-tree suitable for packaging”.

Put simply, craft-parts gives developers consistent tools to fetch, build, and prepare software from any ecosystem for packaging into various formats.

Lifecycle stages #

Every part has a minimum of four lifecycle stages:

PULL: source code or binary artifacts, along with dependencies are pulled from various sources
BUILD: software is built automatically by a plugin, or a set of custom steps defined by the developer
STAGE: select outputs from the BUILD phase are copied to a unified staging area for all parts
PRIME: files from the staging area are copied to the priming area for use in the final artifact.

The STAGE and PRIME steps are similar, except that PRIME only happens after all parts of the build are staged. Additionally, STAGE provides the opportunity for parts to build/supply dependencies for other parts, but that might not be required in the final artifact.

Lifecycle in the CLI #

The lifecycle stages aren’t just in the build recipe, they’re also first-class citizens in each craft’s CLI, thanks to the craft-cli library. This ensures a consistent command-line experience across all craft tools.

Take the following examples:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Run the full process including PULL, BUILD, STAGE, PRIME and then pack the final artifact
snapcraft pack
charmcraft pack
rockcraft pack

# Run the process up to the end of the STAGE step
rockcraft stage

# Run the process up to the PRIME step
charmcraft prime

This design feature supports a smoother iterative development and debugging workflow for building and testing software artifacts.

Part definition #

The parts of a build vary in complexity - some require two-three trivial lines, others require detailed specification of dependencies, build flags, environment variables and steps. The best way to understand the flexibility of this system is by looking at some examples.

First, consider this (annotated) example from my icloudpd snap:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


icloudpd:
 # Use the 'python' plugin to build the
 # software. This takes care of identifying
 # Python package dependencies, building the wheel
 # and ensuring the project's dependencies are staged
 # appropriately.
 plugin: python
 # Fetch the project from Github, using the tag the matches
 # the version of the project.
 source: https://github.com/icloud-photos-downloader/icloud_photos_downloader
 source-tag: v$SNAPCRAFT_PROJECT_VERSION
 source-type: git

This spec is everything required to fetch, build and stage the important bits required to run the software - in this case a Python wheel and its dependencies.

Some projects might require more set up, perhaps an additional package is required or a specific version of a dependency is needed. Let’s take a look at a slightly more complex example taken from my zinc-k8s-operator project:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


kube-log-runner:
 # Use the 'go' plugin to build the software.
 plugin: go
 # Fetch the source code from Git at the 'v0.17.0' tag.
 source: https://github.com/kubernetes/release
 source-type: git
 source-tag: v0.17.8
 # Change to the specified sub-directory for the build.
 source-subdir: images/build/go-runner
 # Install the following snaps in the build environment.
 build-snaps:
 - go/1.20/stable
 # Set the following environment variables in the build
 # environment.
 build-environment:
 - CGO_ENABLED: 0
 - GOOS: linux

This instructs rockcraft to fetch a Git repository at a particular tag, change into the sub-directory images/build/go-runner, then build the software using the go plugin. It also specifies that the build required the go snap from the 1.20/stable track, and sets some environment variables. That’s a lot of result for not much YAML. The end result of this is a single binary that’s “staged” and ready to be placed (in this case) into a Rock (Canonical’s name for OCI images).

And the best part: this exact definition can be used in a rockcraft.yaml when building a Rock, a snapcraft.yaml when building a Snap, a charmcraft.yaml when building a Charm, etc.

The plugin system is extensive: at the time of writing there are 22 supported plugins, including go, maven, uv, meson and more. If your build system of choice isn’t supported you can specify manual steps, giving you as much flexibility as you need:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


wasi-sdk:
 # There is no appropriate plugin for this part, so set
 # it to 'nil' and we'll specify our own build process
 # using 'override-build'.
 plugin: nil
 # In this recipe, a previous part named 'clang' is
 # required to build before attempting to build this
 # part.
 after:
 - clang
 # Specify any `apt` packages required in the build
 # environment.
 build-packages:
 - wget
 # Set some environment variables for the build
 # environment.
 build-environment:
 - WASI_BRANCH: "15"
 - WASI_RELEASE: "15.0"
 # Define how to pull the software manually.
 override-pull: |
 ROOT=https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-$WASI_BRANCH
 wget $ROOT/wasi-sysroot-$WASI_RELEASE.tar.gz
 wget $ROOT/libclang_rt.builtins-wasm32-wasi-$WASI_RELEASE.tar.gz
 # Define how to 'build' the software manually
 override-build: |
 craftctl default
 tar -C $CRAFT_STAGE -xf wasi-sysroot-$WASI_RELEASE.tar.gz
 tar -C $CRAFT_STAGE/usr/lib/clang/* -xf libclang_rt.builtins-wasm32-wasi-$WASI_RELEASE.tar.gz
 # Don't prime anything for inclusion in the
 # final artifact; this part is only used for
 # another part's build process.
 override-prime: ''

Here, multiple stages of the lifecycle are overridden using override-build, override-pull and override-stage, and we see craftctl default for the first time, which instructs snapcraft to do whatever it would have done prior being overridden, but allows the developer to provide additional steps either before or after the default actions.

Isolated build environments #

Even once a recipe for building software is defined, preparing machines to build software can be painful. Different major versions of the same OS might have varying package availability, your team might run completely different operating systems, and you might have limited image availability in your CI environment.

The crafts solve this with build “backends”. Currently the crafts can use LXD or Multipass to create isolated build environments, which makes it work nicely on Linux, macOS and Windows. This functionality is handled automatically by the crafts through the craft-providers library. The craft-providers library provides uniform interfaces for creating build environments, configuring base images and executing builds.

This means if you can run snapcraft pack on your machine, your teammates can also run the same command without worrying about installing the right dependencies or polluting their machines with software and temporary files that might result from the build.

One of my favourite features of this setup is the ability to drop into a shell inside the build environment automatically on a few different conditions:

1
2
3
4
5
6


# Drop into a shell if any part of the build fails.
snapcraft pack --debug
# Drop into a shell after the build stage.
rockcraft build --shell-after
# Drop to a shell in lieu of the prime stage.
snapcraft prime --shell

This makes troubleshooting a failing build much simpler, while allowing the developer to maintain a clean separation between the build environment and their local machine. Should the build environment ever become polluted, or otherwise difficult to work with, you can always start from a clean slate with snapcraft|rockcraft|charmcraft clean. Each build machine is constructed using a cached build-base, which contains all the baseline packages required by the craft - so recreating the build environment for a specific package only requires that base to be cloned and augmented with project specific concerns - making the process faster.

Saving space #

When packaging any kind of software, a common concern is the size of the artifact. This might be because you’re building an OCI-image that is pulled thousands of times a day as part of a major SaaS deployment, or maybe it’s a Snap for an embedded device running Ubuntu Core with a limited flash. In the container world, “distroless” became a popular way to solve this problem - essentially popularising the practice of shipping the barest minimum in a container image, eschewing much of the traditional Unix FHS.

The parts mechanism has provided a way of “filtering” what is staged or primed into a final artifact from the start, which already gave developers autonomy to choose exactly what went into their builds.

In addition to this, Canonical built “chisel”, which extends the distroless concept beyond containers to any kind of artifact. With chisel, developers can slice out just the binaries, libraries, and configuration files they need from the Ubuntu Archive, enabling ultra-small packages without losing the robustness of Ubuntu’s ecosystem.

We later launched Chiseled JRE containers, and there are numerous other Rocks that utilise chisel to provide a balance between shipping tiny container images, while benefiting from the huge selection and quality of software in the Ubuntu Archive.

Because the crafts are all built on a common platform, they now all have the ability to use “slices” from chisel-releases, which enables a greater range of use-cases where artifact size is a primary concern. Slices are community maintained, and specified in simple to understand YAML files. You can see the list of available slices for the most recent Ubuntu release (25.04 Plucky Puffin) on GitHub, and further documentation on slices and how they’re used in the Chisel docs.

Multi-architecture builds #

Ubuntu supports six major architectures at the time of writing (amd64, arm64, armhf, ppc64le, s390x, riscv64), and all of our crafts have first-class support for each of them. This functionality is provided primarily by the craft-platforms library, and supported by the craft-grammar library, which enables more complex definitions where builds may have different steps or requirements for different architectures.

At a high-level, each artifact defines which architectures or platforms it is built for, and which it is built on. These are often, but not always, the same. For example:

1
2


platforms:
 amd64:

This is shorthand for “build the project on amd64 for amd64”, but in a different example taken from a charmcraft.yaml

1
2
3
4


platforms:
 all:
 build-on: [amd64]
 build-for: [all]

In this case the software is built on amd64, but can run on any of the supported architectures - this can happen with all-Python wheels, bash scripts and other interpreted languages which don’t link platform-specific libraries.

In some build processes, the process or dependencies might differ per-architecture, which is where craft-grammar comes in, enabling expressions such as (taken from GitHub):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


fit-image:
 # ...
 build-packages:
 # ...
 - wget
 - libjson-c-dev:${CRAFT_ARCH_BUILD_FOR}
 - libcryptsetup-dev:${CRAFT_ARCH_BUILD_FOR}
 # Only use the following build packages when building for armhf
 - to armhf:
 - binutils-arm-linux-gnueabi
 - gcc-arm-linux-gnueabihf
 - pkgconf:armhf
 # When building for arm64, use a different set
 - to arm64:
 # Dependencies for building *for* arm64 *on* amd64!
 - on amd64:
 - gcc-aarch64-linux-gnu
 - pkgconf:arm64
 - on arm64:
 - gcc

Being able to define how to build on different architectures is only half of the battle, though. It’s one thing to define how to build software on an s390x machine but few developers have mainframes handy to actually run the build! This is where the crafts’ remote-build capability comes in. The remote-build command sends builds to Canonical’s build farm, which has native support for all of Ubuntu’s supported architectures. This is built into all of our crafts, and is triggered with snapcraft remote-build, rockcraft remote-build, etc.

Remote builds are a lifeline for publishers and communities who need to reach a larger audience, but can’t necessarily get their own build farm together. One example of this is Snapcrafters, a community-driven organisation that packages popular software as Snaps, who use remote-build to drive multi-architecture builds from GitHub Actions as part of their publishing workflow (as seen here and here for example).

Unified testing framework #

Testing is often the missing piece in build tools: developers are forced to rely on separate CI systems or ad-hoc scripts to verify their artifacts. To close this gap, we’re introducing a unified test sub-command in the crafts.

We recently added the test sub-command to our crafts as an experimental (for now!) feature. Under the hood, craft test will introduce a new lifecycle stage (TEST). The enables packagers of any artifact type to specify how that artifact should be tested using a common framework across artifact types.

Craft’s testing capability is powered by spread, a convenient full-system task distribution system. Spread was built to simplify the massive number of integration tests run for the snapd project. It enables developers to specify tests in a simple language, and distribute them concurrently to any infrastructure they have available.

This enables a developer to define tests and test infrastructure, and make it trivial to run the same tests locally, or remotely on cloud infrastructure. This can really speed up the development process - preventing developers from needing to wait on CI runners to spin up and test their code while iterating, they can run the very same integration tests locally using craft test.

There are lots of fine details to spread, and the team is working on artifact-specific abstractions for the crafts that will make testing delightful. Imagine maintaining the Snap for a GUI application, and being able to enact the following workflow:

1
2
3
4
5
6
7
8
9


# Pull the repository
git clone https://github.com/some-gui-app/snap && cd snap
# Make some changes, perhaps fix a bug
vim snap/snapcraft.yaml
# Build the snap, and run the integration tests.
# These tests might include spinning up a headless
# graphical VM, which actually installs and runs
# the snap, and interacts with it
snapcraft test

By integrating a common testing tool into the build tooling, the Starcraft team will be able to curate unique testing experiences for each kind of artifact. A snap might need a headless graphical VM, where an OCI-image simply requires a container runtime, but the spread underpinnings allow a common test-definition language for each.

There are a couple of examples of this in the wild already:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# Install charmcraft
sudo snap install --classic charmcraft
# Clone the repo
git clone https://github.com/jnsgruk/zinc-k8s-operator
cd zinc-k8s-operator
# List the available tests
charmcraft test --list lxd:
# Run the integration testing suite, spinning up
# a small VM, inside which is a full Kubernetes
# instance, with a Juju controller bootstrapped.
# From here the charm will be deployed and tested to
# ensure it's integrations with the observability
# stack and ingress charms are functioning correctly.
charmcraft test -v lxd:ubuntu-24.04:tests/spread/observability-relations:juju_3_6

The test above is powered by this spread.yaml, and this test definition. With a little bit of work, it’s also possible to integrate spread with GitHub matrix actions, giving you one GitHub job per spread test - as seen here.

You can see a similar example in our PostgreSQL Snap test suite, and we’ll be adding more and more of this kind of test across our Rock, Snap, Charm, Image and Deb portfolio.

There is work to do, but I’m really excited about bringing a common testing framework to the crafts which should make the testing of all kinds of artifacts more consistent and easier to integrate across teams and systems.

Crafting the crafts #

As the portfolio expanded from snapcraft, to charmcraft, to rockcraft and is now expanding further to debcraft and imagecraft it was clear that we’d need a way to make it easy to build crafts for different artifacts, while being rigorous about consistency across the tools. A couple of years ago, the team built the craft-application base library, which now forms the foundation of all our crafts.

The craft-application library combines many of the existing libraries that were in use across the crafts (listed below), providing a consistent base upon which artifact-specific logic can be built. The allows craft developers to spend less time implementing CLI details, parts lifecycles and store interactions, and more time on curating a great experience for the maintainers of their artifact type.

For the curious, craft-application builds upon the following libraries:

craft-archives: manages interactions with apt package repositories
craft-cli: CLI client builder that follows the Canonical’s CLI guidelines
craft-parts: obtain, process, and organize data sources into deployment-ready filesystems.
craft-grammar: advanced description grammar for parts
craft-providers: interface for instantiating and executing builds for a variety of target environments
craft-platforms: manage target platforms and architectures for craft applications
craft-store: manage interactions with Canonical’s software stores
craft-artifacts: pack artifacts for craft applications

Examples and docs #

Before I leave you, I wanted to reference a few *craft.yaml examples, and link to the documentation for each of the crafts, where you’ll find the canonical (little c!) truth on each tool.

You can find documentation for the crafts below:

And some example recipes:

Snap: icloudpd - snapcraft.yaml
Snap: parca-agent - snapcraft.yaml
Snap: signal-desktop - snapcraft.yaml
Charm: ubuntu-manpages-operator - charmcraft.yaml
Rock: grafana - rockcraft.yaml
Rock: temporal-server - rockcraft.yaml

Summary #

The craft ecosystem provides developers with a rigorous, consistent and pleasant experience for building many kinds of artifacts. At the moment, we support Snaps, Rocks and Charms but we’re actively developing crafts for Debian packages, cloud images and more.The basic build process, parts ecosystem and foundations of the crafts are “battle tested” at this point, and I’m excited to see how the experimental craft test commands shape up across the crafts.

One of the killer features for the crafts is the ability to reuse part definitions across different artifacts - which makes the pay off for learning the parts language very high - it’s a skill you’ll be able to use to build Snaps, Rocks, Charms, VM Images and soon Debs!

If I look at ecosystems like Debian, where tooling like autopkgtest is the standard, I think debcraft test will offer an intuitive entrypoint and encourage more testing, and the same is true of Snaps, both graphical and command-line.

That’s all for now!

Introducing Debcrafters

Mon, 30 Jun 2025 00:00:00 +0000

This article was originally posted on the Ubuntu Discourse, and is reposted here. I welcome comments and further discussion in that thread.

Earlier this year, Canonical’s Ubuntu Engineering organisation gained a new team, seeded with some of our most prolific contributors to Ubuntu. Debcrafters is a new team dedicated to the maintenance of the Ubuntu Archive.

The team’s primary goal is to maintain the health of the Ubuntu Archive, but its unique construction aims to attract a broad range of Linux distribution expertise; contributors to distributions like Debian, Arch Linux, NixOS and others are encouraged to join the team, and will even get paid to contribute one day per week to those projects to foster learning and idea sharing

Bootstrapping the team #

The Debcrafters team is a global team. We have a squad in the Americas, a squad in EMEA and will have a squad in APAC. At present, we’ve staffed the AMER and EMEA teams with existing Canonical employees from our Foundations, Desktop, Server and Public Cloud teams. Each team currently has a manager, and four engineers.

The team comprises Debian Developers, Stable Release Updates (SRU) team members and archive administrators, and began working together for the first time at our recent Engineering Sprint in Frankfurt held in early May 2025.

Mission #

The Debcrafters’ primary mission is to maintain the health of the Ubuntu Archive.

This team will take the lead on syncing & merging packages from Debian, reviewing proposed migration issues, upstreaming Ubuntu deltas, and take ownership of major transitions such as upgrades to glibc and past examples such as the t64 and python3 transitions.

They’ll manage the scheduling, triggering and reporting on archive test rebuilds which we conduct when making major changes to critical packages. We did this when we enabled frame pointers by default, and when we switched coreutils to the uutils implementation in Ubuntu 25.10.

They’ll be responsible for the evolution and maintenance of the autopkgtest infrastructure for Ubuntu, as well as taking an instrumental role in introducing more distro-scale integration tests.

They’ll work on improving the reporting and dashboarding of the Ubuntu Archive, its contributors and status, as well as taking a broader interest in shaping the tools we use to build and shape Ubuntu.

What sets this team apart from the likes of Desktop, Server and Foundations is the range of packages they will work on. Members of the Debcrafters team will move thousands of packages every cycle - many of which they will not be intimately familiar with, but will use their growing distro maintenance and packaging skills to perform maintenance where there is no other clear or present owner.

Tools & processes #

One of the key goals in my first post was to modernise the contribution experience for Ubuntu Developers by focusing on tools and processes.

The Debian project recently adopted tag2upload, which allows Debian Developers to use git-debpush to push a signed git tag when uploading packages. While we’re not following that exact path, we share many of the same goals and intentions.

For some time Ubuntu Developers have been able to use git-ubuntu as part of their development workflow, which aims to provide “unified git-based workflows for the development of Ubuntu source packages”. This project brought us closer to our desired experience, but still needs work to achieve our complete vision. I’d like to put more emphasis on the experience we provide for testing packages, as well as signing, uploading and releasing packages.

In the coming weeks our Starcraft team (responsible for Snapcraft, Rockcraft, Charmcraft) will begin prototyping debcraft, which will (in time) become the de facto method for creating, testing and uploading packages to the Ubuntu archive.

The first prototype of debcraft will focus on unifying the current workflow adopted by most Ubuntu Developers at Canonical. It will wrap existing tools (such as git-ubuntu, lintian, autopkgtest) to provide familiar, streamlined commands such as debcraft pack, debcraft lint and debcraft test. Uploading packages, and a more native “craft” experience for constructing packages will come later.

Details will make their way into the new Ubuntu Project Docs throughout the course of the 25.10 Questing Quokka cycle, including the newly renovated “Ubuntu Packaging Guide”, which will aim to provide a “one ring to rule them all” approach to documenting how to package software for Ubuntu.

Attracting contributors #

While the team has been seeded with seasoned Ubuntu contributors, one of the primary goals of the team is to grow the contributor base across generations.

One of the sub-teams is currently leading the roll out of a new contributor journey that will soon be publicly available. This process lays out the journey from complete beginner to “Core Dev”, stopping off at “Package Maintainer”, “Package Set Maintainer”, “MOTU”, etc. along the way. The process also aims to help candidates prepare for Developer Membership Board interviews.

Whether you’re a junior engineer just graduating from University, or you’re a seasoned Linux contributor elsewhere in the Linux ecosystem, the Debcrafters team is an excellent place to learn software packaging skills and contribute to the world’s most deployed Linux distribution.

Contribution beyond Ubuntu #

The Debcrafters’ primary commitment is to Ubuntu, but we recognise the enormous value in collaborating with other distributions. Many of the hard lessons I’ve personally learned resulted from contributing to NixOS and building Snaps. Packaging is a complex and ever-changing discipline, and other distributions are facing many of the complex problems we are - often with different or novel approaches to solving them.

In recognition of this, we’re actively seeking maintainers from other distributions - be that Debian, Arch, NixOS, Guix, Fedora, Universal Blue or any other - packaging and distribution engineering skills are often common across distributions, and we believe that Ubuntu can benefit from broader perspectives, while contributing back to the wider ecosystem of distributions in the process.

The Debcrafters must spend the majority of their work time on Ubuntu, but they will be encouraged to spend a day per week contributing to other distributions to gain understanding, and bring fresh perspectives to Ubuntu (and the reverse, hopefully!). This will be structured as a literal day per week, agreed with the team management - for example “I work on NixOS on Tuesdays”.

Summary #

Canonical has launched a new team, the Debcrafters, who are dedicated to maintaining the very core of Ubuntu: the archive. This team has a global footprint, and deep expertise in software packaging drawn from across the Linux ecosystem. They’ll lead transitions, improve tooling improvements and strengthen our distribution testing infrastructure.

Whether you’re an experienced Debian Developer, a maintainer from another Linux distribution or a new engineer starting your career in open source, Debcrafters offers a unique opportunity to learn, grow, and contribute to the world’s most widely deployed Linux distribution.

From NixOS to Ubuntu

Tue, 24 Jun 2025 00:00:00 +0000

Following my appointment as VP Engineering for Ubuntu, I moved all of my machines from NixOS to Ubuntu. Being responsible for decisions that affect millions of Ubuntu users comes with, in my opinion, the obligation to use the product and live with those decisions myself.

Following years of running Arch Linux and NixOS, I imagined this would be uncomfortable, but was pleasantly surprised. In this post, I’ll outline my setup and a new philosophy for how I configure my machines.

Last year, I wrote in detail about my setup. Consider this post a “diff” on what’s changed since.

Recap: Why NixOS? #

NixOS affords a seemingly endless selection of applications, desktop environments and when combined with Home Manager it provides a consistent way to manage the configuration of almost all aspects of a system.

People have argued that this is overkill, and results in needlessly complex configurations that produce difficult to read error messages, and make a system more difficult to troubleshoot.

While I had some troubles adopting NixOS, I consistently felt that the positives outweighed the negatives: I loved being able to overlay individual packages and tweak fundamentals of the system in a machine specific way. I also liked being able to trivially reuse configuration for app and hardware configuration across my machines in a single flake, which remains (to my amazement) one of my most popular GitHub repositories.

As I planned my move to Ubuntu, I decided to change the way I used my computer to avoid fighting my machine, and any feeling that I could be missing out on the ~~complexity~~ flexibility I’d become so used to.

Adopting “JFUI” #

The guiding principle for my new way of thinking is JFUI: Just F*cking Use It!

The primary motivation behind JFUI is to pick applications for which the defaults are close enough to my preferences, then use them with as little (or no) configuration as possible.

By employing this principle, I should spend less of my time updating my configuration files as things change, and spend less time obsessing over every last theme detail for each application on my machine.

I’ve spent years collecting repositories full of dotfiles, which contains the ~2500 lines of configuration and scripts I used prior to moving to NixOS. As of today, the most recent commit in my Nix flake, cloc reports:

---------------------------------------------------
Language files blank comment code
---------------------------------------------------
Nix 113 401 232 3666
Bourne Again Shell 3 25 30 95
Bourne Shell 1 17 0 83
Markdown 1 19 1 81
YAML 4 8 3 51
diff 1 1 7 6
---------------------------------------------------
SUM: 123 471 273 3982
---------------------------------------------------

And that seems like a lot to me! Yet this number omits my Visual Studio Code configuration (a further 200 lines of JSON), various browser configurations, etc.

So I challenged myself to set up my Ubuntu machines with as little configuration as possible and to choose apps with better defaults.

Desktop Environment #

It’d been a long time since I’d used a computer regularly without a tiling window manager. I switched to sway in 2019, and to Hyprland in 2023.

I’ve built a lot of muscle memory by using a consistent keymap across both environments. I use a 57" ultrawide monitor, and the idea of using it without tiling seemed like total anarchy to me.

But it didn’t take me long to adapt to using GNOME again. Despite abstaining from it for years, I’ve always appreciated the visual design of GNOME and often used their apps as part of my tiling experience (Files, Papers, Loupe).

Features such as the keyring, light/dark mode switching worked for me under Sway/Hyprland, but it always required complex, fragile configuration. The out of the box experience for configuring WiFi networks and Bluetooth devices feels modern, and like a part of the OS, rather than a kit of parts. In my latter few months of running Hyprland, I struggled more with consistent theming and stability as Hyprland evolved separately from the themes and apps I liked.

I don’t attribute any blame for this; it’s a natural side effect of combining lots of independent and often complex parts from across the Linux desktop ecosystem, and I was consciously running pre 1.0 software knowing there would be issues because I enjoyed the overall experience.

GNOME Extensions #

This is the area where I violated JFUI the most, though some of it is temporary as I get further away from my existing workflow. I disable the Ubuntu dock, desktop icons, app indicators and Ubuntu tiling assistant. While the tiling assistant was a great step up on what came before it, it fell short of what I needed to manage windows on my large display.

I took inspiration from Omakub, which is where I discovered Tactile - a GNOME extension for tiling windows to a custom on-screen grid using the keyboard. I’ve found this to be invaluable, and easily the best window management experience for GNOME. I’ve customised the grid to the following ratios (using gsettings in a script):

I’ve grown to like Space Bar - it provides desktop workspaces akin to those in Sway/Hyprland which enabled me to use the muscle memory I’d developed over years. As I’ve progressed, I think I could live without Space Bar and use the native workspace features in GNOME, so I’ll experiment with that soon.

The final two are somewhat simpler: Emoji Copy (mapped to Super + E) and Clipboard History (mapped to Super + V), functions that were previously enabled by wofi with a couple of plugins and lots of configuration.

Editors #

I spend a lot of time in an editor. As I wrote last year, I’ve been using neovim and Visual Studio Code for some years. I never “managed” Visual Studio Code with NixOS because I always found their settings sync to be quite sufficient.

My nvim config wasn’t too complicated - though it is made to look simpler because Home Manager abstracts away the details of managing plugins. I hadn’t taken the time to set up language server support, code completion or other creature comforts that one might expect from an editor in 2025.

I’d dabbled before with Helix, a modal command-line editor not dissimilar from vim. Helix comes with a lot more out of the box: it supports the Language Server Protocol (LSP), has many built in colour schemes, supports fuzzy finding files/buffers, project wide search and more. The keymap took some practice, but my entire configuration amounts to 8 lines including some blank lines and provides many more modern features.

1
2
3
4
5
6
7
8
9


theme = "catppuccin_macchiato"

[editor]
cursorline = true
text-width = 100
rulers = [100]

[editor.lsp]
display-inlay-hints = true

There is a separate file with some LSP configuration, but only a few more lines. Once I got this set up, I spent about 8 weeks using just Helix to make sure that the keymap and operating model were sufficiently burned in to my mind. I’ve been really impressed - Helix is fast, the default features are great and I don’t anticipate returning to vim.

On the desktop side, I’d was intrigued by Zed. I don’t always feel the need for a desktop editor, but there are projects that I prefer working on in a more graphical environment. I last tried Zed about 18 months ago and found it a little too sparse on features, but things have really evolved since then and there is an encouraging rate of change on the project. I’ve been using it most days for the last 4-5 months, and while it still lacks some of the polish (and features) of Visual Studio Code, it’s significantly lighter on resources, and I’m much happier with the defaults (details below)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


{
 "ui_font_size": 18,
 "buffer_font_size": 18,
 "theme": {
 "mode": "system",
 "light": "Catppuccin Latte",
 "dark": "Catppuccin Macchiato"
 },
 "buffer_font_family": "MesloLGMDZ Nerd Font Mono",
 "ui_font_family": ".SystemUIFont",
 "base_keymap": "SublimeText",
}

Terminal #

I switched to zsh around 2013 and have used it every day since: starting with the infamous Oh My Zsh, powerlevel9k, and subsequently powerlevel10k, and finally settling on Starship with a couple of zsh plugins such as zsh-autosuggestions and zsh-syntax-highlighting.

My zsh config had become quite complex - an area in which NixOS/Home Manager really helped by providing (mostly)neat abstractions for plugins and starship integration, but the equivalent configuration on Ubuntu would have been 100s of lines of zsh, notwithstanding the need to load plugins in the right order, etc.

Most of the configuration I was doing with zsh was to imitate fish. I’d tried fish in the past, but reverted when I couldn’t use sudo !!, or /some/command $! and other tricks. I’d been following fish’s rewrite and decided to give it another go, and have stuck with it since. It fits my JFUI mantra perfectly, leaving me with just a few lines of config for essentially identical functionality:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


set fish_greeting ""

fish_add_path ~/.nix-profile/bin
fish_add_path ~/.local/bin
fish_add_path ~/.cargo/bin
fish_add_path ~/go/bin
fish_add_path ~/scripts

if status is-interactive
 starship init fish | source
 fzf --fish | source
 atuin init fish --disable-up-arrow | source
end

set -gx EDITOR hx
set -gx SUDO_EDITOR hx

I still occasionally fall foul of typing sudo !!, but overall I’ve found fish to be an excellent interactive shell replacement. In particular, the native tab completion support is leagues ahead of anything I ever managed to configure with zsh.

I moved from Alacritty to Ghostty, which has been absolutely excellent. It’s wicked fast, it has my favourite colour scheme built in, and I just love the way @mitchellh has set the project up for success in the long term. It also fits in nicely with my minimally configured applications, with excellent defaults out of the box.

1
2
3
4
5
6


theme = catppuccin-macchiato
font-family = MesloLGMDZ Nerd Font Mono
font-size = 14
window-padding-x = 10
window-padding-y = 10
window-decoration = false

Finally: tmux, which I had been happily using for many years, but slowly collecting configuration for. I decided to give Zellij a try, and haven’t looked back. It also has a hellishly complicated configuration in my case 😉:

1
2
3


theme "catppuccin-macchiato"
default_layout "compact"
show_startup_tips false

Software Availability #

I always felt incredibly spoiled by the vast availability of software for NixOS, and even more so that the contribution model was so simple that I was able to augment to collection myself.

On Ubuntu, I use Snaps wherever possible and fall back to archive and installing software with apt where it makes sense. If neither of those have what I need, I use the Nix package manager, which works well on Ubuntu. While some of the “additional” software might be installable using go install, or cargo install, or other package managers like brew, flakpak, npm, etc., I wanted to keep things as simple as possible, so if it’s not available from Ubuntu-native sources I get it from nixpkgs.

I’ve included what I’m actually getting from the Snap store (63 snaps), and from nixpkgs (21 packages) below.

Installed snaps

❯ snap list | tail -n+2
agendrr 0.1.1 3 latest/stable jnsgruk* -
astral-uv 0.7.13 627 latest/stable lengau classic
audacity 3.7.1 1208 latest/candidate snapcrafters* -
bare 1.0 5 latest/stable canonical** base
bitwarden 2025.5.1 140 latest/stable bitwarden** -
charmcraft 3.4.6 6672 latest/stable canonical** classic
code 18e3a1ec 197 latest/stable vscode** classic
core 16-2.61.4-20250508 17212 latest/stable canonical** core
core18 20250523 2887 latest/stable canonical** base
core20 20250526 2599 latest/stable canonical** base
core22 20250528 2010 latest/stable canonical** base
core24 20250526 1006 latest/stable canonical** base
desktop-security-center 0+git.f7ad73a 59 1/stable/… canonical** -
discord 0.0.98 243 latest/candidate snapcrafters* -
docker 28.1.1+1 3265 latest/stable canonical** -
dotrun 1.4.8 85 latest/stable canonicalwebteam -
ffmpeg-2404 7.1.1 75 latest/stable snapcrafters* -
firefox 139.0.4-1 6316 latest/stable/… mozilla** -
firmware-updater 0+git.22198be 167 1/stable/… canonical** -
ghstat 0.4.1 91 latest/stable jnsgruk* -
ght 1.11.7 110 latest/edge tbmb -
gimp 3.0.4 525 latest/stable snapcrafters* -
gnome-3-28-1804 3.28.0-19-g98f9e67.98f9e67 198 latest/stable canonical** -
gnome-3-34-1804 0+git.3556cb3 93 latest/stable canonical** -
gnome-42-2204 0+git.38ea591 202 latest/stable/… canonical** -
gnome-46-2404 0+git.d9f8bf6-sdk0+git.c8a281c 90 latest/stable canonical** -
go 1.24.4 10907 latest/stable canonical** classic
gopls 0.19.0 1089 latest/stable alexmurray* classic
goreleaser 2.10.2 1060 latest/stable caarlos0 classic
gtk-common-themes 0.1-81-g442e511 1535 latest/stable/… canonical** -
helix 25.01.1 91 latest/stable lauren-brock classic
icloudpd 1.28.1 12 latest/stable jnsgruk* -
jhack 0.4.4.0.13 461 latest/stable ppasotti -
jq 1.5+dfsg-1 6 latest/stable mvo* -
juju 3.6.7 31266 3/stable canonical** -
kubectl 1.33.2 3609 latest/stable canonical** classic
lxd 5.21.3-c5ae129 33110 5.21/stable canonical** -
mattermost-desktop 5.12.1 789 latest/stable snapcrafters* -
mesa-2404 24.2.8-snap183 887 latest/stable canonical** -
multipass 1.15.1 14535 latest/stable canonical** -
node 22.16.0 10226 22/stable iojs** classic
obsidian 1.8.10 47 latest/stable obsidianmd classic
pinta 3.0.1 56 latest/stable james-carroll* -
prompting-client 0+git.d542a5d 104 1/stable/… canonical** -
rambox 2.4.1 44 latest/stable ramboxapp** -
rockcraft 1.12.0 3367 latest/stable canonical** classic
ruff 0.11.13 1377 latest/stable lengau -
rustup 1.27.1 1471 latest/stable canonical** classic
shellcheck v0.10.0 1725 latest/stable koalaman -
shfmt 3.5.1 33 latest/stable ankushpathak -
signal-desktop 7.58.0 799 latest/candidate snapcrafters* -
snap-store 0+git.90575829 1270 2/stable/… canonical** -
snapcraft 8.9.4 15082 latest/stable canonical** classic
snapd 2.68.5 24718 latest/stable canonical** snapd
snapd-desktop-integration 0.9 253 latest/stable/… canonical** -
sublime-merge 2102 95 latest/stable snapcrafters* classic
thonny 4.1.7 239 latest/stable sameersharma2006 -
thunderbird 128.11.1esr-1 737 latest/stable canonical** -
todoist 9.17.0 1340 latest/stable doist** -
typescript-language-server 4.3.4 211 latest/stable alexmurray* -
yazi shipped 293 latest/stable sxyazi classic
yq v4.44.5 2634 latest/stable mikefarah -
zellij 0.42.2 41 latest/stable dominz88 classic

Installed Nix packages

❯ nix profile list | grep -Po "Name:[ ]*\K.+\$"
atuin
bash-language-server
cargo-udeps
deadnix
fzf
gh
gofumpt
nil
nixd
nixfmt-rfc-style
prettier
pyright
python-lsp-server
spread
starship
statix
taplo
terraform-ls
typos-lsp
vscode-langservers-extracted
yaml-language-server

I continue to find Nix development shells useful, and even use one to develop this site.

Configuration Management #

In the past I’ve used all kinds of scripts, Ansible playbooks and dotfile managers to solve this problem, and its a problem that was solved very elegantly by NixOS/Home Manager. I experimented with Home Manager on Ubuntu to manage dotfiles and configuration, but found the experience had more rough edges than I would like, and didn’t really adhere to my JFUI principles.

I’ve settled on a directory full of idempotent, single-purpose scripts which get executed by a wrapper named provision. This is somewhat inspired by Omakub, but without the menus and configuration they supply to allow their users some customisation. The scripts install packages, write configuration with tools like gsettings and symlink configuration into place where needed:

❯ ls
app-1password dev-python tool-agendrr
app-chrome dev-rust tool-atuin
app-flameshot font-meslo tool-gearlever
app-ghostty hw-yubikey tool-gh
app-obsidian kara-audioengine tool-git
app-pinta kara-backup tool-helix
app-rambox kara-data-disk tool-junction
app-signal kara-hiring-automation tool-lxd
app-sublime-merge kara-hiring-reports tool-multipass
app-thunderbird provision tool-podman
app-todoist system-flatpak tool-spread
configs system-fs tool-starship
dev-charms system-gnome tool-syncthing
dev-containers system-gnome-extensions tool-tailscale
dev-go system-nix tool-zed
dev-node system-shell tool-zellij

An example of a specific script, such as tool-zellij:

1
2
3
4
5
6


#!/usr/bin/env bash
set -e
DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
sudo snap install --classic zellij
mkdir -p "${HOME}/.config/zellij"
ln -sf "${DIR}/configs/zellij/config.kdl" "${HOME}/.config/zellij/config.kdl"

The provision script is similarly simple (and naive):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


#!/usr/bin/env bash
set -ex
DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )

sudo apt-get update
sudo apt-get upgrade -y

categories="system hw dev tool app font $(hostname)"
for c in $categories; do
 for x in ${c}-*; do
 source $x
 done
done

This is just about satisfactory. I’ve got into the habit of only ever installing software by creating the relevant script and configuration. If I don’t need it to persist, the work gets done in an ephemeral LXD container/VM then thrown away.

I’m working on a better solution for this named miso, short for “Make It So”. This a homegrown, multi-host configuration management tool for my machines that draws inspiration from cloud-init, home-manager, terraform and a few others. I’ll write about that in a future post when the code is a little more complete, but there is an example of an early configuration format I’m targeting available as a gist.

Everything in the linked config file is currently implemented and working with a decent suite of integration tests - but I’ve got a lot of tidying to do with error handling and such before I release it.

So What About Nix? #

I thoroughly enjoyed my adventures with Nix, and I consider having learned how to package software with Nix and use the available tooling to manage servers, desktops and development shells to have been incredibly worthwhile.

Even if I were to never touch Nix again, the general packaging and distribution engineering skills I learned have been invaluable, and I’m grateful to everyone who helped me on that journey through Matrix chats, Pull Requests and Mastodon interactions.

I remain active on the ~35 packages in nixpkgs for which I’m the maintainer. I continue to use Nix for development shells, CI and for certain packages on my Ubuntu machines. I have archived my flake for now because it’s not being maintained, but I’ve left it there in case there are any patterns that might be useful for others.

Summary #

Ubuntu has been very stable on my desktop, server and both of my laptops. I’ve enjoyed the level of integration and polish that comes with no effort in the desktop environment, and managing less configuration has been a freeing experience - even if some of my apps no longer have matching themes 😱.

Provisioning and configuration management are less structured and more cumbersome in Ubuntu, but that has driven me to build my own tool which was good fun, and gave me a nice challenge to solve through my journey learning Rust! I hope to learn from the project in a way that helps inform the development of Ubuntu itself in future releases.

If you’re into numbers, here are the stats for my renewed “config” directory, which contains all of the text-based configuration & scripts for my machines:

----------------------------------------------------------
Language files blank comment code
----------------------------------------------------------
Bourne Again Shell 47 154 81 413
TOML 3 13 1 53
Fish Shell 1 13 3 38
JSON 1 0 0 35
----------------------------------------------------------
SUM: 52 180 85 539
----------------------------------------------------------

From 3982 lines, to 539 lines, and much of that could be reduced if it wasn’t for the slightly repetitive nature of maintaining separate, idempotent scripts. Not bad.

As much as I feared the transition, my journey back to Ubuntu has been very enjoyable. I’m not “quitting Nix” or “over it”, but at least for now I’m enjoying a less complex existence with my personal computers.

Thanks for reading!

Supercharging Ubuntu Releases: Monthly Snapshots & Automation

Thu, 29 May 2025 00:00:00 +0000

This article was originally posted on the Ubuntu Discourse, and is reposted here. I welcome comments and further discussion in that thread.

Introduction #

Ubuntu has shipped on a predictable, six-month cadence for two decades. Twenty years ago, the idea of releasing an entire distribution every six months was considered forward looking, bold and even difficult. Things have changed since then: software engineering has evolved as a practice, and the advent of both rolling-release distributions like Arch Linux, and more recently image-based immutable distributions such as Universal Blue have meant that other projects with similar goals have adopted vastly different release models with some desirable properties.

My goal over the coming months is to build a release process that takes advantage of modern release engineering practices, while retaining the resilience and stability of our six-monthly releases. We’ll introduce significantly more automated testing, and ensure that the release process is transparent, repeatable and executable in a much shorter and well-known timeframe with little to no human intervention.

This journey will also create space for better system-wide testing, earlier detection of regressions, and a more productive collaboration with our community.

Monthly Snapshot Releases #

Starting in May 2025, we’re introducing monthly snapshot releases for Ubuntu.

Ubuntu is not “moving to monthly releases” or adopting a rolling release model; we’re committed to our six-monthly releases with a Long Term Support (LTS) release every two years. That doesn’t mean that our release process should be exempt from the same scrutiny that the rest of our engineering processes are subject to.

Today the Ubuntu Release process is the product of twenty years of evolution: it safeguards Ubuntu releases with a wealth of checks and balances, but is a largely manual process requiring significant human involvement.

The Ubuntu Release Team is a crowd of seasoned Ubuntu veterans who have been steadily releasing Ubuntu for many years. Many of this team are community members, some are or have been employed by Canonical in the past. More recently we have established the Canonical Ubuntu Release Management Team - a relatively new team at Canonical who’ll be collaborating with the Ubuntu Release Team to develop the new process.

To aid the Canonical team in their understanding of the existing processes, and the immovable requirements that sit beneath it, we’re introducing monthly snapshot releases for Ubuntu. These will not be fully-fledged releases of Ubuntu, but rather curated, testable milestones from our development stream. For the 25.10 (Questing Quokka) cycle, you can expect the following release schedule:

May 29, 2025: Questing Quokka - Snapshot 1
June 26, 2025: Questing Quokka - Snapshot 2
July 31, 2025: Questing Quokka - Snapshot 3
August 28, 2025: Questing Quokka - Snapshot 4
September 18, 2025: Questing Quokka - Beta
October 9, 2025: Questing Quokka - Final Release

This doesn’t mean you’ll start seeing Ubuntu versions off the six-month cadence. There will be no Ubuntu 25.07 or 25.08, etc. The monthly snapshots are exactly that: a snapshot of the development of Ubuntu 25.10. Snapshots are not meant for production use, but will help the release team move away from deep institutional knowledge, and toward clean well-documented automated workflows that are transparent, repeatable and testable.

With our current model, failure modes are not detected until they’re urgent and blocking an imminent release. The team conducts rigorous retrospectives on each release, but in my opinion it’s hard to meaningfully evolve such a process when it’s only exercised every six months. The monthly snapshots will create opportunities for us to test, understand and improve the process.

Embracing Automation #

One of the most valuable outcomes of this journey will be the opportunity to automate more of the process, freeing up time for the team to focus on more strategic tasks. Releasing a distribution is a complex process requiring coordination across architectures, images, mirrors, websites, testing infrastructure and even partner agreements. This also makes it hard to place a traditional CI tool at the heart of the process. As much as I like Github Actions, I think we’d quickly get lost trying to release Ubuntu with such a system, notwithstanding the fact that we’d lose control of the underlying infrastructure that releases Ubuntu.

I’ve been exploring the world of Durable Execution, which according to restate.dev is:

the practice of making code execution persistent, so that services recover automatically from crashes and restore the results of already completed operations and code blocks without re-executing them.

At Canonical, we’ve adopted Temporal in a few of our products and in many of our business processes. Temporal is a durable execution product that enables developers to solve complex distributed problems, but without being deep distributed systems experts. It’s a framework for composing tasks into workflows, with first-class primitives for dealing with failures, retries, exponential back-off and other concepts that enable the build of long-running complex workflows.

Having spent some time with Temporal myself, and watched other teams adopt it, I think it’s a great fit for engineering our next-generation release process. I want our engineers to focus on the logic of the release process, not the infrastructure behind it, and Temporal should enable them to do just that. The Temporal homepage sums it up nicely:

Write code as if failure doesn’t exist

Temporal workflows and activities can be written in many languages - and particularly in Python and Go. My expectation is that Go will prove to be an excellent fit for our process: it’s a fast and productive language that specialises in concurrency and asynchronous network operations, and has a powerful standard library containing much of the functionality we’ll need to build our new release process.

To take an overly simplistic view of how I expect this to go: we’ll take our existing release checklist, write a Go function for each step with some tests, and compose them together into one or more Temporal workflows that represent the full release process. This will take time, but this approach will enable us to incrementally demonstrate progress toward a fully-automated process over the coming cycles.

By making this move, not only will we make the process quicker, but also more observable, testable, reliable and easier to understand for everyone, not just the release team.

Improving Test Coverage #

One area I’d like to improve as a side-effect of this work is more full-system integration testing. Packages in the Ubuntu archive generally enjoy good coverage through a suite of autopkgtest tests, and there are numerous other places where integration tests are run on Ubuntu. With our traditional six-monthly cadence, full end-to-end testing of ISOs and the installer typically ramps up close to release time when changes are fewer (and riskier) and time is short.

With the introduction of monthly snapshots, we can integrate installer testing, full-disk encryption testing, graphical application testing and more as a regular, automated part of the release pipeline - not just as part of the development pipeline of each individual package. This means we should catch regressions earlier and surface more edge cases to be resolved before release.

One of the most important parts of increasing our testing culture is to make it clear where and how to contribute tests to Ubuntu. The easier we make it to write and contribute tests, the more tests we’re likely to add to the suite. We’re doing some work on this in parallel which will likely turn into a blog post of its own in the coming months.

In our current process, we have a heroic group of volunteers who kindly spend hours on our behalf testing the various flavours - exercising all the possible install paths and validating that what is about to be published is fit for purpose. I’d like to ensure that our volunteers’ time is spent as productively and rewardingly as possible, and I think we can automate much of this testing and allow them to focus on the more complex and nuanced aspects of each release, and raise the quality of Ubuntu across all the flavours.

What’s Next? #

We’re starting by modeling the current release process as it is. Once we’ve validated our assumptions about the current process, we’ll layer in improvements by reducing manual gates, parallelising independent steps, introducing more testing, and exercising the process each month to test (and measure) any improvements we’ve made.

My ultimate goal is a release system that’s incredibly “boring”: transparent, predictable, observable, and easy to reason about (even when things go wrong).

The new, fully-automated process will likely take several months to complete. When we think we’re done, we’ll do a release that runs both processes in parallel to ensure we get the outcome we expect before finally sunsetting the old process.

We’ll be building this work in the open (and hiring!) so if you’ve used Temporal in similar contexts, or are curious about contributing to this effort, we’d love to hear from you.

Until next time!

Adopting sudo-rs By Default in Ubuntu 25.10

Tue, 06 May 2025 00:00:00 +0000

This article was originally posted on the Ubuntu Discourse, and is reposted here. I welcome comments and further discussion in that thread.

Introduction #

Following on from Carefully But Purposefully Oxidising Ubuntu, Ubuntu will be the first major Linux distribution to adopt sudo-rs as the default implementation of sudo, in partnership with the Trifecta Tech Foundation

The change will be effective from the release of Ubuntu 25.10. You can see the Trifecta Tech Foundation’s announcement here.

What is `sudo-rs`? #

sudo-rs is a reimplementation of the traditional sudo tool, written in Rust. It’s being developed by the Trifecta Tech Foundation (TTF), a nonprofit focused on building secure, open source infrastructure components. The project is part of the Trifecta Tech Foundation’s Privilege Boundary initiative, which aims to handle privilege escalation with memory-safe alternatives.

The sudo command has long served as the defacto means of privilege escalation on Linux. As described in the original post, Rust provides strong guarantees against certain classes of memory-safety issues, which is pivotal for components at the privilege boundary.

The sudo-rs team is collaborating with Todd Miller, who’s maintained the original sudo for over thirty years. sudo-rs should not be considered a fork in the road, but rather a handshake across generations of secure systems. Throughout the development of sudo-rs, the TTF team have also made contributions to enhance the original sudo implementation.

The sudo-rs project is designed to be a drop in replacement for the original tool. For the vast majority of users, the upgrade should be completely transparent to their workflow. That said, sudo-rs is a not a “blind” reimplementation. The developers are taking a “less is more” approach. This means that some features of the original sudo may not be reimplemented if they serve only niche, or more recently considered “outdated” practices.

Erik Jonkers, Chair of the Trifecta Tech Foundation explains:

While no piece of software - in any language - is flawless, we believe the transition to Rust in systems programming is a vital step forward, it is very exciting to see Ubuntu committing to sudo-rs and taking the lead in moving the needle.

Sponsoring Mainstream Adoption #

Leading the mainstream adoption of a replacement to such a universally understood tool comes with responsibility. Before committing to ship sudo-rs in Ubuntu 26.04 LTS, we’ll test the transition in Ubuntu 25.10. We’re also sponsoring the development of some specific items, which has manifested as Milestone 5 in the upstream project:

Coarse-grained shell escape prevention (NOEXEC) on Linux (See PR #1073)
The ability to control AppArmor profiles (First PR #1067)
A sudoedit implementation
Support for Linux Kernels older than version 5.9

The final item may seem out of place, but because Ubuntu 20.04 LTS is still in support, without this work there could be situations where sudo fails to function if, for example, a 26.04 LTS OCI container was run on a 20.04 LTS host!

The team have also already begun work on ensuring that the test-suite is as compatible as possible with Ubuntu, to ensure any issues are caught early.

This isn’t just about shipping a new binary. It’s about setting a direction. We’re not abandoning C, or even rewriting all the utilities ourselves, but by choosing to replace one of the most security-critical tools in the system with a memory-safe alternative, we’re making a statement: resilience and sustainability are not optional in the future of open infrastructure.

Progress on `coreutils` #

Since the initial announcement, we’ve been working hard to more clearly define a plan for the migration to uutils coreutils in 25.10 and beyond. Similarly to our engagement with the Trifecta Tech Foundation, we’re also sponsoring the uutils project to ensure that some key gaps are closed before we ship 25.10. The sponsorship will primarily cover the development of SELinux support for common commands such as mv, ls, cp, etc.

The first step toward developing SELinux support was to add support for automated testing in Github Actions, since then the maintainers have begun work on the actual implementation.

The other feature we’re sponsoring is internationalisation support. At present, some of the utility implementations (such as sort) have an incomplete understanding of locales, and therefore may yield unexpected results. We expect that these two features should land in time for us to ship in 25.10, and we’ll continue to work with the uutils project throughout the 26.04 LTS cycle to close any remaining gaps we identify in the interim release.

One of the major concerns outlined in Julian’s post is about binary size. We’ve got a few tricks we can play here to get the size down, and there is already some conversation started upstream in Debian on how that might be achieved. There are also security implications, such as AppArmor’s lack of support for multi-call binaries. We’re currently working with the respective upstreams to discuss addressing this systematically, through in the interim we may need to build small wrapper binaries to enable compatibility with existing AppArmor profiles from the start.

Migration Mechanics #

Julian Klode posted recently on the Ubuntu Discourse outlining the packaging plan that will enable us both to migrate transparently to uutils coreutils, but also provide a convenient means for users to opt-out and switch back to GNU coreutils if they wish, or if they identify a gap in the new implementation. I expect this will be rare, but we want to make sure it’s as easy as possible to revert, and will be documenting this in detail before release.

Replacing coreutils isn’t as simple as swapping binaries. As an Essential package, its replacement must work immediately upon unpacking without relying on maintainer scripts, and without conflicting files across packages. To solve this, we’re introducing new coreutils-from-uutils and coreutils-from-gnu packages, as well as coreutils-from itself. For all the gory details, see the Discourse post!

The packaging work required to switch to sudo-rs is somewhat less complicated than with coreutils. The package is already available in Ubuntu (which you can still test on Ubuntu 24.04, 24.10 and 25.04 with oxidizr!), but unlike coreutils, sudo is not an Essential package, so we’ll be able to make use of the Debian alternatives system for the transition.

Summary #

Things are progressing nicely. We’ve established strong, productive relationships and are sponsoring work upstream to make these transitions viable.

We’ve got a strategy for migrating the default implementation of coreutils and sudo in Ubuntu 25.10 which will enable a seamless revert in cases where that is desired. While sudo-rs will be the default in 25.10, the original sudo will remain available for users who need it, and we’ll be gathering feedback to ensure a smooth transition before the 26.04 LTS.

Additionally, we’ve begun investigating the feasibility of providing SequoiaPGP and using it in APT instead of GnuPG. SequoiaPGP is a new OpenPGP library with a focus on safety and correctness, written in Rust. The GnuPG maintainers have recently forked the OpenPGP standard and are no longer compliant with it. Sequoia provides a modern alternative to GnuPG with strict behavior, and is already used in various other systems. More details to follow!

Stay tuned!

Revitalising Ubuntu Project Documentation

Tue, 01 Apr 2025 00:00:00 +0000

This article was originally posted on the Ubuntu Discourse, and is reposted here. I welcome comments and further discussion in that thread.

Introduction #

Back in February I wrote some thoughts on Ubuntu’s documentation and its role within the community. For a mostly-online software community, documentation is one of our most critical forms of communication.

In the last two years there has been lots of focus on the technical aspects of our documentation (how-to guides, tutorials, etc.), but I’d like to focus more on what I’m calling the “Ubuntu Project Documentation” over the coming months.

Documentation isn’t only about technical how-to guides and tutorials, nor is it only about troubleshooting or satisfying particular use-cases. Our documentation can set the tone for the project, give a means for the community to state an intent, and guide both current and future contributors in their daily work.

Ubuntu has a lot of documentation, most of which has grown organically over the last 20 years, but it’s not always easy to find or understand. Our documentation should illuminate and inspire a path to contribution. It should provide direction and clarity on complex issues, reference on technology and past decisions, and precision in the execution of process.

Our project documentation should detail what makes Ubuntu happen. How are decisions made? What are the teams contributing to Ubuntu? How are those teams appointed? What are their responsibilities? If you’re on the Main Inclusion Review (MIR) team and you’re assigned a package to review, what steps should you take? How does package sponsorship work, and who should you contact if you’re stuck? How are the Access Control Lists (ACLs) updated for packages and package sets, and who can make those changes? What does the journey look like from first time package bug-fixer to Ubuntu Core Developer?

These are all examples of questions that we, the collective conscious of Ubuntu, know the answers to, yet it is still difficult to find up-to-date answers to these questions, often requiring input from some of our busiest and most knowledgeable contributors to settle discussions and answer basic queries.

If a potential contributor identifies a bug in a package, there should be one authoritative source of information on where the package source can be located, how it can be pulled, built and tested, and how to work with a sponsor to land changes. Such a process is satisfying for contributors, making it more likely they’ll stay engaged, and therefore benefits the distribution’s longevity and sustainability.

Answering questions and mentoring people will remain a central part of our community’s role, but many of the first questions asked could be serviced by better documentation.

The Challenge #

Much of the required content already exists. The venerable Ubuntu Wiki was the go-to destination for such documentation, but has become outdated both technologically and in the content it serves. This degradation gained pace as we diversified the number of destinations that documentation could live: the Wiki, Discourse, Github, Launchpad, etc.

The Ubuntu Community team have made significant efforts over the past months to centralise the documentation for membership, our code of conduct and project governance. I also called out the renewed Stable Release Update (SRU) documentation in my first post for having made its first steps toward a new and improved structure.

These examples prove we have all the skills we need to write excellent documentation. Throughout the 25.10 cycle, I intend to put some focus on this, consolidating as much of our content into modern formats as possible, thereby making it as accessible as possible.

In doing this work, I hope to:

Illustrate contributor journeys across disciplines
Create resilience in the project by reducing the “bus factor”
Increase the accessibility and ergonomics of our documentation
Enable more efficient, asynchronous collaboration on a wide range of tasks

End Goal #

To quote the Canonical.com page on Documentation Practice:

we have embarked on a comprehensive, long-term project to transform documentation. Our aim is to create and maintain documentation product and practice that will represent a standard of excellence. We want documentation to be the best it possibly can be.

At the heart of this mission Diátaxis: a way of thinking about documentation. Diátaxis “prescribes approaches to content, architecture and form that emerge from a systematic approach to understanding the needs of documentation users”.

You’ll have seen Diátaxis in use across many of our product documentation pages: the Juju docs, the MAAS docs, the Pebble docs, the Rockcraft docs and many more.

Most of those existing sites are specific - they document a particular product or ecosystem which neatly scopes the documentation structure, but the Diátaxis framework can also be used to bring structure, precision and clarity to the documentation of the Ubuntu project as a whole.

Earlier this month I surveyed the various documentation sites in use by Canonical and the Ubuntu Community, and settled on three common themes around which we will structure our renewed project documentation:

Governance: in which membership, code of conduct, team structures, communication practices, delegation, mission, software licensing and 3rd-party software guidelines will be documented.
Develop Ubuntu: documentation for current and aspiring Ubuntu developers, including how to package software for Ubuntu, how to merge packages from Debian, how to sponsor packages, how to use git-ubuntu and conduct “+1 Maintenance”, etc.
Archive Administration: the nuts and bolts of managing Ubuntu’s prolific software repositories: how to manage seeds, configure phased updates, conduct an MIR, run an SRU process, etc.

These categories were not immediately obvious, and they’re not necessarily mutually exclusive, but they fell out quite naturally when trying to logically organise our existing content.

During the process, I came up with this rough sketch:

This illustrates how multiple categories of documentation from different corners of Ubuntu might come together in a single landing page. To give an idea of how we might further break down existing content by type, then category:

This may not be the final structure, but it’s indicative of how we can use Diátaxis to break down large documentation premises into smaller, more digestible and more ergonomic pieces.

The Plan #

During the Ubuntu 25.10 cycle, we’ll be dedicating two of our Technical Authors to make this happen. One of these authors has been largely responsible for overhauling the Ubuntu Server docs, but both are very familiar with Diátaxis and the tooling we using to deliver documentation.

Throughout this process, we’ll likely come across outdated, poorly reviewed or incorrect documentation, but as we work through the process of consolidating, we can note where this has happened and get it on our backlog to fix.

Perhaps we’ll find items which lend themselves to inclusion in the Canonical Open Documentation Academy, or maybe we’ll need to reach out to some of our less active community members for clarification, but once the structure is in place we’ll at least have a place to collaborate.

Once the transition is complete, there will be an authoritative source for project documentation that is easy to navigate, easy to contribute to and with a well-defined review process that encourages progress over gatekeeping.

Summary #

Documentation is the backbone of a thriving open-source community, guiding contributors, setting expectations, and ensuring long-term sustainability.

While Ubuntu has extensive documentation, much of it is scattered, outdated, or difficult to navigate. By leveraging the Diátaxis framework, we aim to bring structure, clarity, and accessibility to Ubuntu Project Documentation. Our focus will be on governance, development, and archive administration, ensuring that key processes and responsibilities are well-documented and easy to follow.

With dedicated technical authors and community collaboration, the Ubuntu 25.10 cycle will mark a significant step toward making our documentation searchable, structured, and sustainable.

I hope this effort will empower contributors, reduce reliance on institutional knowledge, and create a more resilient project for the next generation of Ubuntu developers and users.

Carefully But Purposefully Oxidising Ubuntu

Wed, 12 Mar 2025 00:00:00 +0000

This article was originally posted on the Ubuntu Discourse, and is reposted here. I welcome comments and further discussion in that thread.

Introduction #

Last month I published Engineering Ubuntu For The Next 20 Years, which outlines four key themes for how I intend to evolve Ubuntu in the coming years. In this post, I’ll focus on “Modernisation”. There are many areas we could look to modernise in Ubuntu: we could focus on the graphical shell experience, the virtualisation stack, core system utilities, default shell utilities, etc.

Over the years, projects like GNU Coreutils have been instrumental in shaping the Unix-like experience that Ubuntu and other Linux distributions ship to millions of users. According to the GNU website:

The GNU Core Utilities are the basic file, shell and text manipulation utilities of the GNU operating system. These are the core utilities which are expected to exist on every operating system.

This package provides utilities which have become synonymous with Linux to many - the likes of ls, cp, and mv. In recent years, there has been an effort to reimplement this suite of tools in Rust, with the goal of reaching 100% compatibility with the existing tools. Similar projects, like sudo-rs, aim to replace key security-critical utilities with more modern, memory-safe alternatives.

Starting with Ubuntu 25.10, my goal is to adopt some of these modern implementations as the default. My immediate goal is to make uutils’ coreutils implementation the default in Ubuntu 25.10, and subsequently in our next Long Term Support (LTS) release, Ubuntu 26.04 LTS, if the conditions are right.

But… why? #

Performance is a frequently cited rationale for “Rewrite it in Rust” projects. While performance is high on my list of priorities, it’s not the primary driver behind this change. These utilities are at the heart of the distribution - and it’s the enhanced resilience and safety that is more easily achieved with Rust ports that are most attractive to me.

The Rust language, its type system and its borrow checker (and its community!) work together to encourage developers to write safe, sound, resilient software. With added safety comes an increase in security guarantees, and with an increase in security comes an increase in overall resilience of the system - and where better to start than with the foundational tools that build the distribution?

I recently read an article about targeting foundational software with Rust in 2025. Among other things, the article asserts that “foundational software needs performance, reliability — and productivity”. If foundational software fails, so do all of the other layers built on top. If foundational packages have performance bottlenecks, they become a floor on the performance achievable by the layers above.

Ubuntu powers millions of devices around the world, from servers in your data centre, to safety critical systems in autonomous systems, so it behooves us to be absolutely certain we’re shipping the most resilient and trustworthy software we can.

There are lots of ways to achieve this: we can provide long term support for projects like Kubernetes, we can assure the code we write, and we can strive to achieve compliance with safety-centric standards, but another is by shipping software with the values of safety, soundness, correctness and resilience at their core.

That’s not to throw shade on the existing implementations, of course. Many of these tools have been stable for many years, quietly improving performance and fixing bugs. A lovely side benefit of working on newer implementations, is that it sometimes facilitates improvements in the original upstream projects, too!

I’ve written about my desire to increase the number of Ubuntu contributors, and I think projects like this will help. Rust may present a steeper learning curve than C in some ways, but by providing such a strong framework around the use of memory it also lowers the chances that a contributor accidentally commits potentially unsafe code.

Introducing `oxidizr` #

I did my homework before writing this post. I wanted to see how easy it was for me to live with these newer implementations and get a sense of their readiness for prime-time within the distribution. I also wanted a means of toggling between implementations so that I could easily switch back should I run into incompatibilities - and so oxidizr was born!

oxidizr is a command-line utility for managing system experiments that replace traditional Unix utilities with modern Rust-based alternatives on Ubuntu systems.

The oxidizr utility enables you to quickly swap in and out newer implementations of certain packages with relatively low risk. It has the notion of Experiments, where each experiment is a package that already exists in the archive that can be swapped in as an alternative to the default.

Version 1.0.0 supports the following experiments:

How does it work? #

Each experiment is subtly different since the paths of the utilities being replaced vary, but the process for enabling an experiment is generally:

Install the alternative package (e.g. apt install rust-coreutils)
For each binary shipped in the new package:
- Lookup the default path for that utility (e.g which date)
- Back up that file (e.g. cp /usr/bin/date /usr/bin/.date.oxidizr.bak)
- Symlink the new implementation in place (e.g. ln -s /usr/bin/coreutils /usr/bin/date)

There is also the facility to “disable” an experiment, which does the reverse of the sequence above:

For each binary shipped in the new package:
- Lookup the default path for the utility (e.g which date)
- Check for and restore any backed up versions (e.g cp /usr/bin/.date.oxidizr.bak /usr/bin/date)
Uninstall the package (e.g. apt remove rust-coreutils)

Thereby returning the system back to its original state! The tool is covered by a suite of integration tests which illustrate this behaviour which you can find on Github

Get started #

⚠️ WARNING ⚠️: oxidizr is an experimental tool to play with alternatives to foundational system utilities. It may cause a loss of data, or prevent your system from booting, so use with caution!

There are a couple of ways to get oxidizr on your system. If you already use cargo, you can do the following:

1

cargo install --git https://github.com/jnsgruk/oxidizr

Otherwise, you can download and install binary releases from Github:

1
2


# Download version 1.0.0 and extract to /usr/bin/oxidizr
curl -sL "https://github.com/jnsgruk/oxidizr/releases/download/v1.0.0/oxidizr_Linux_$(uname -m).tar.gz" | sudo tar -xvzf - -C /usr/bin oxidizr

Once installed you can invoke oxidizr to selectively enable/disable experiments. The default set of experiments in v1.0.0 is rust-coreutils and sudo-rs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Enable default experiments
sudo oxidizr enable
# Disable default experiments
sudo oxidizr disable
# Enable just coreutils
sudo oxidizr enable --experiments coreutils
# Enable all experiments without prompting with debug logging enabled
sudo oxidizr enable --all --yes -v
# Disable all experiments without prompting
sudo oxidizr disable --all --yes

The tool should work on all versions of Ubuntu after 24.04 LTS - though the diffutils experiment is only available from Ubuntu 24.10 onward.

The tool itself is stable and well covered with unit and integration tests, but nonetheless I’d urge you to start with a test virtual machine or a machine that isn’t your production workstation or server! I’ve been running the coreutils and sudo-rs experiments for around 2 weeks now on my Ubuntu 24.10 machines and haven’t had many issues (more on that below…).

How to Help #

If you’re interested in helping out on this mission, then I’d encourage you to play with the packages, either by installing them yourself or using oxidizr. Reply to the Discourse post with your experiences, file bugs and perhaps even dedicate some time to the relevant upstream projects to help with resolving bugs, implementing features or improving documentation, depending on your skill set.

You can also join us to discuss on our Matrix instance.

Next Steps #

Earlier this week, I met with @sylvestre to discuss my proposal to make uutils coreutils the default in Ubuntu 25.10. I was pleased to hear that he feels the project is ready for that level of exposure, so now we just need to work out the specifics. The Ubuntu Foundations team is already working up a plan for next cycle.

There will certainly be a few rough edges we’ll need to work out. In my testing, for example, the only incompatibility I’ve come across is that the update-initramfs script for Ubuntu uses cp -Z to preserve selinux labels when copying files. The cp, mv and ls commands from uutils don’t yet support the -Z flag, but I think we’ve worked out a way to unblock that work going forward, both in the upstream and in the next release of Ubuntu.

I’m going to do some more digging on sudo-rs over the coming weeks, with a view to assessing a similar transition.

Summary #

I’m really excited to see so much investment in the foundational utilities behind Linux. The uutils project seems to be picking up speed after their recent appearance at FOSDEM 2025, with efforts ongoing to rework procps, util-linux and more.

The sudo-rs project is now maintained by the Trifecta Tech Foundation, who are focused on “open infrastructure software in the public interest” . Their zlib-rs recently released v0.4.2, which appears to now be the fastest API-compatible zlib implementation. They’re also behind the Pendulum Project and ntpd-rs for memory-safe time synchronisation.

With Ubuntu, we’re in a position to drive awareness and adoption of these modern equivalents by making them either trivially available, or the default implementation for the world’s most deployed Linux distribution.

We will need to do so carefully, and be willing to scale back on the ambition where appropriate to avoid diluting the promise of stability and reliability that the Ubuntu LTS releases have become known for, but I’m confident that we can make progress on these topics over the coming months.

Own Your Calendar

Mon, 03 Mar 2025 00:00:00 +0000

Introduction #

Over the past few years, I’ve come to a series of realisations about how I can get the most from my calendar, or rather how to get the most from my time at work, while preventing work from spilling out into my personal life too much.

Staying on top of my calendar has helped me manage my time better. It helps me get more done, and set reasonable work-life boundaries. Calendar management helps my colleagues understand how I spend my time, what I’m working on, and when it might be best to book my time.

Bound Your Day #

The first step toward taking control of your calendar is to bound your day. In more conventional workplaces where everyone commutes into an office, working patterns are frequently more predictable. With a remote, globally distributed workforce things are harder to predict. Your 9-5 might be in the middle of the night for others, and either you or your colleagues might have flexible working arrangements that make their work hours less obvious.

One way to help colleagues book time with you smoothly is by clearly indicating when your day starts, when it ends, and when you’ll break for lunch. Your calendar tool might do a good job of this already.

Here’s an indicative example of how I set up my calendar:

Some of you are raising your eyebrow over taking an hour for lunch every day, but hear me out. This time is not just used for the critical act of feeding yourself, but gives you the chance to get some fresh air, rest your eyes and give your brain a chance to focus on something else for a while. How many times have you solved a complicated problem in the shower? This is the same thing.

Indicating to your colleagues through status messages that you’re on a break on Slack/Mattermost/Teams/whatever can also be helpful.

There may be occasions when people need to get hold of you right now, and you should let them know how to do that. For me, the bat phone of choice is Signal. No matter how much I silence my work email notifications, or use Do Not Disturb on Mattermost, Signal is the only app that never gets silenced because it’s what I use to communicate with those closest to me - so my colleagues know to use that if they need to, and that frees me up to concentrate on the day and not get too distracted by a stream of instant messages and emails.

Make Space For Focus #

There are two kinds of focus time I need in my day: one to triage emails and messages, respond to “new tasks” and another to progress “planned work”.

Like many others, I get a lot of email and instant messages. Because of the global nature of my company, these tend to keep coming overnight. I like to set aside 30 minutes every morning assigned to “Catch Up”. In that time, I respond to emails and messages I’ve received overnight, review my schedule for the rest of the day, and set up Obsidian for note-taking (see How I Computer in 2024). As a side-effect of this, I’m always up to date by the time I start my first meeting each day.

For planned work, I set aside at least one hour per day. This is often neglected, leading to people becoming overwhelmed by meetings: group meetings, team 1:1s, daily/weekly/fortnightly/monthly rituals, leadership meetings, etc. Before long, there’s no slack in their day to actually think about their work.

An hour a day is a minimum, but at least by blocking it out you guarantee that minimum. Depending on your role and responsibilities, these slots may occupy more of less of your week.

These occur at the same times each week in my calendar where possible, making it easier for my colleagues to plan meetings when they need to. I label these slots with what I’ll focus on in that time. In my calendar these are recurring events named “Focus Time”, then every Monday during my “Catch Up” slot, I figure out what’s important to get done in that week, and label each slot with what I plan to work on. You might label a slot with the Jira ticket you’ll tackle, the specific Professional Development activity you’re planning, the document you’ll review or maybe it’s a Pair Programming session with a colleague.

In my experience, this practice helps my colleagues understand what I’m working on, but on busy days it also helps me avoid wasting time deciding what to work on when I become free between meetings - my calendar tells me!

Once I’ve conducted my Monday “Catch Up” each week, the focus blocks in my calendar look more like this:

In the above example, I’ve extended the Thursday slot to get something specific done. I avoid planning contiguous focus blocks for more than 2 or 3 hours. Much beyond that, and most people will begin to lose focus, become less effective and more frustrated, ultimately getting less done. I find I’m better taking a break to do something else, then coming back to the task later with fresh eyes.

Plan Regular Meetings #

I try to be deliberate about the meetings I attend. In most roles there are meetings which are an inescapable reality. These might include:

Team 1:1s
Leadership Syncs
Project Reviews
Planning Meetings
Mentoring/Coaching
Demos

These will vary in length and cadence. Try to schedule your most demanding meetings at the time of day when you’re at your best; for me this translates into scheduling most of my team 1:1s in the morning (timezones permitting!).

If you have regular fortnightly meetings, try to ensure that there is something scheduled on the “off weeks” at the same time. This will prevent accidentally scheduling a weekly meeting in a slot where you’re already committed fortnightly - a surprisingly easy mistake.

I find it helpful to label meetings with their cadence. This may not help day-to-day, but can help in reviewing how you spend your time (more on that later).

In my current role, I attend a business review every six weeks. In these weeks, Tuesday and Thursday afternoons are consumed by business review activity. As a result, I make sure there are no recurring meetings in those times to avoid a scramble to re-arrange them all every six weeks. In weeks where I don’t have Business Reviews, these are great spots to use for focus time, interviews, and other ad-hoc meetings.

Maintain Blank Space #

This might be the most important point in the whole post: ensure there is blank space in your calendar every day outside of your regular planned events.

This can be hard to achieve, and may not manifest in you actually having blank space by the time each day starts, but if your work week is already 100% booked with regular events and planned work, how will you respond to unplanned events? Unexpected customer meetings? Without any blank space in your calendar, you’re destined to spend an unhealthy amount of time worrying about or rearranging your calendar, and struggle to get things done.

I’m able to be more present in meetings, more able to ignore notifications and distractions when I know there will be time in the day to get around to them. If there is never any space for these tasks, I’m easily distracted by staying on top of message notifications, emails, etc.

A good test of this is to look forward 2-3 weeks from now in your calendar. What does it look like? If there is no blank space in your calendar, then start reviewing regular commitments and get it back under control.

I recently re-read It Doesn’t Have To Be Crazy At Work, having read it first some years ago. While there are some minor points that don’t quite resonate with me, the general principle that we should stop glorifying packed schedules and competing with our colleagues to be the busiest or the most overworked is absolutely spot on.

If you’re in a leadership position, this is not just for your benefit but for everyone around you too. People look to leadership for their example; organisations and teams mimic the habits of their leaders over time. If you’re in work from 6am to 9pm every day and your schedule is constantly back-to-back, there’s a good chance others will copy, glorify, and expect those behaviours of others (yes, even if you tell them they don’t have to, and you don’t expect it of them, and…).

Bookable Placeholders #

You may have recurring tasks each week that are less predictable. This could be interviews, customer calls, or anything where you expect to perform a certain number each week, but can’t always predict the exact timings in advance.

My approach is to use blank space for this, but an alternate approach is to add placeholders in your calendar that indicate when you would prefer for those events to be scheduled. This can save time and round-trips via email/instant message, and reduce the number of times you’re booked for something at a time that doesn’t suit you.

Most calendaring tools will allow you to mark time like this in your calendar without it showing you as “Busy”. This will take some experimentation while you understand your average weekly commitment and when those events are most likely to occur, but might look something like this:

Review Regularly #

Even if you do all of the above, your calendar will fill up over time. You’ll get invited to the next important monthly review meeting, perhaps collect an extra report or two, take responsibility for a major project, or be asked to mentor a colleague. There are countless ways in which your blank space can get eaten up, and when your calendar becomes cluttered, it’s likely to create a higher mental load which distracts you from the work you really need to get done.

Pick a cadence on which you review all of your regular engagements for necessity, length and frequency. At Canonical we have company roadmap sprints every 3 months, and I’ve found that to be a useful cadence (and reminder) to review my calendar. After each sprint, I spend one of my focus blocks staring at my calendar trying to work out which planned meetings are still useful and effective, and which I’m going to either stop attending or reduce the frequency of.

A couple of examples where I’ll “trim the fat”:

Group meetings that have grown too large over time, and become less effective.
Mentoring or coaching engagements where significant progress has been made, and the frequency can be reduced. Perhaps the relationship with the person you’re mentoring is good enough that you can revert to ad-hoc scheduling when they need assistance.
Placeholders that are going unused week-to-week.

Remote Work and Flexibility #

One of the advantages of remote work is the flexibility it affords employees, but it’s important to be respectful of that privilege. The foundations of most effective workplaces are trust and respect. A structured and well-planned calendar doesn’t mean reducing flexibility - in many cases a structured calendar can enable more flexibility.

If you’re fortunate enough to work remotely, then you should take advantage of that, but it’s also important to stay accountable and make it predictable where you can. If you go to the gym every Wednesday at 10am, then put that in your calendar and make it clear on your calendar where that time is made up. That way, your colleagues can plan around it.

I like mountain biking and I find the winter (in the UK) pretty miserable. I go mountain biking on a Monday afternoon from 1300-1600, but then I work on a Monday from 1900-2200. This actually fits in with my role - a number of my reports are in the US or Australia, and returning to work on a Monday evening means I can do our 1:1s in their timezone without asking them to work late/early.

For me the benefit is actually seeing daylight for a decent length of time in the middle of the day. If the weather is poor and I don’t feel like biking, I do something else on Monday afternoons, but I still keep that commitment to my US/APAC colleagues in the evening. This consistency makes it easy to plan around, but still gives me plenty of chances to go biking.

Tips & Tricks #

There are some other tricks that can help you get the most from your calendar:

Buffer Time: Give yourself a few minutes between meetings. Whether it be to get up and stretch your legs, grab a drink or whatever. I configure Google Calendar to default to 25 minute and 50 minute meetings, rather than 30 minutes and 60 minutes. On the days I’m disciplined enough to stick to that, it gives me a few valuable minutes between meetings to pee!
Colour Coding/Emojis: I used to dismiss this, but have come to really appreciate it. Distinctively marking items in your calendar can help you subconsciously prepare for what’s coming, as well as help you see where you spend the majority of your time. I use dark blue regular meetings, yellow “People” meetings, purple for focus blocks, green for external meetings and orange for business/commerical review calls.
Task Management: When I’m asked to do something which will take me more than 5 minutes, I generally put that task in my calendar. If I’m asked to review a pull request and I think it’ll take me 30 minutes, I create a 30 minute event in some of the blank space named “Review PR #113”. This ensures I get the time I need, and the requester gets to see when I’ve planned the work.

Summary #

This post summarises a collection of lessons I’ve learned over time. Each of these have incrementally improved my time management at work. I feel more productive and less stressed, which means I find it easier to switch off from work in the evening and at weekend.

My final note is on flexibility. Most of this article describes principles - ideas you should try to implement. In reality, collaboration and calendar management across teams and timezones is hard. In my experience applying these principles helps account for the “messier” weeks in my work schedule and helps me keep order, but nonetheless you won’t always have total control and you might need to be flexible to accommodate some of your colleagues.

Until next time!

(And thanks to my wife Laura for helping me refine and edit this post!)

Engineering Ubuntu For The Next 20 Years

Tue, 11 Feb 2025 00:00:00 +0000

This article was originally posted on the Ubuntu Discourse, and is reposted here. I welcome comments and further discussion in that thread.

Introduction #

I’ve been a VP Engineering at Canonical for 3 years now, building Juju and our catalog of charms. In the last week of January, I was appointed the VP Engineering for Ubuntu at Canonical, where I will now oversee the Ubuntu Foundations, Server and Desktop teams.

Over the past 20 years, Ubuntu has become synonymous with “Linux” to many people. I fondly remember receiving my first Ubuntu CD in the post, shortly after my own Linux journey began in 2003 with booting Knoppix on a school computer. Throughout my career, Linux and open source have been prominent features that I’m very proud of. In the past few years I’ve made contributions to Ubuntu, Arch Linux, and more recently NixOS.

Ubuntu’s recent 20 year milestone is a timely reminder to pause and reflect on what made Ubuntu so exciting, so successful and so captivating to the Linux community. In 2004, the idea of releasing an operating system every six months was laughed off by many, but has now become the norm. Ubuntu builds upon Debian, aiming to bring the latest and very best open source had to offer to the masses. In the past 10 years, we’ve seen huge shifts in the way software is delivered - the success of large-scale cloud based operations necessitated a shift towards more automated testing, releasing and monitoring, and as the open source community around these projects grew, we had to evolve our ways of thinking, designing and communicating about software.

Four Key Themes #

As I step into this new role, I’ve reflected on how we can steer the engineering efforts behind Ubuntu. I’ve anchored this vision around four themes: Communication, Automation, Process and Modernisation.

Communication #

Communication is a central component of a distributed workforce - whether that workforce is employed by Canonical, members of our community or contributors from our partners. Ubuntu has relied for many years on mailing lists and IRC. These platforms enabled global teams to collaborate for years, and have been invaluable to the community. In 2025 we’re fortunate to have a wealth of communications platforms at our disposal, but we must use these tools strategically to avoid fragmentation.

On Jan 29 2025, the Ubuntu developer mailing list announced that the primary means of communication for Ubuntu developers will be the Ubuntu Community Matrix server. Matrix provides a rich, modern communications medium that is familiar to the next generation of engineers and tinkerers, who will be central to the continued progression of Ubuntu and open source. We’re in good company on Matrix, with many other Linux distributions and projects maintaining a presence on the platform. The recent migration of Ubuntu Forums to the Ubuntu Discourse, further consolidates the range of platforms we use to connect with one another.

To effect much of the change I’m describing in this post, we will need community support. I’ll be encouraging the leads of our internal teams in Ubuntu Foundations, Server and Desktop to be more forthcoming and regular with public updates that will serve two purposes: to share our intentions, progress and dreams for Ubuntu, but also to collaborate on refining our vision, ensuring we deliver a platform that is not just exciting, but relevant for many years to come.

Documentation is a critical form of communication. Our documentation enables our current users, but also illuminates the path for new contributors. Such documentation does exist, but much of it is fragmented across different platforms, duplicated and/or contradictory or simply difficult to find. As a company, and as a community, we must focus on ensuring both existing and potential contributors have access to the information they need on conventions, tools and processes. A good example of where this has already happened is the SRU documentation which was recently rebuilt in line with our documentation practices.

Automation #

Delivering a Linux distribution is a monumental task. With tens of thousands of packages across multiple architectures, the workload can be overwhelming - leaving little room for innovation until the foundational work is done. We’re fortunate to benefit from the diligent work done by the Debian community, yet there is a huge amount of work that goes into each Ubuntu release. One of our primary tasks as a distribution is package maintenance. While some may see this as menial or repetitive, it remains critical to the future of Ubuntu, and is a valuable specialist skill in its own right.

Software packaging is a complex and constantly evolving topic. Ubuntu relies heavily on a blend of Debian packages, and our own Snap packaging format. Debian packaging was revolutionary - responsible for huge advancements in the way we thought about delivering software, but as things have moved on some of those tools and practices are beginning to show their age.

I’d like to focus on enriching our build process with modern ideals and processes for automating the version bumps, testing, performance benchmarking and releasing of packages in the archive. High complexity tasks are error-prone and, without sufficient automation, risk becoming overly dependent on a few skilled individuals. We have the same challenge with Snaps, but they benefit from significantly more modern tooling as a consequence of the observations made about Debian packaging over many years.

The goal of this theme is not just to automate as much as possible (thereby increasing our collective capacity), but also to simplify processes where we can. Much of Ubuntu’s build process *is* automated, but those systems are disparate and often opaque to all but our most experienced contributors.

I’ve been inspired by how the NixOS community manages packaging. Every single package for the distro is represented as text files, in a single Git repository, with a universally observable continuous integration and integration testing pipeline (Hydra) that performs version bumps and simple maintenance tasks semi-autonomously. While this model carries its own challenges, there is something alluring about the transparency and accessibility of the systems that assemble, test and deliver software to their users.

Universal Blue, and by extension Project Bluefin, are recent additions to the Linux ecosystem that benefited from thinking hard about the tooling they use to build their distribution. They’ve centered their process around tools with which their cloud-native audience are already familiar.

My suggestion is not to imitate these projects, rather that the open source community is at its strongest when we collaborate and learn from one another. I think we can take inspiration from those surrounding us, and use that to inform our plans for Ubuntu’s future.

Process #

Process is closely tied to automation, but is frequently viewed negatively in software engineering, carrying connotations of bureaucracy and slowdowns. In my experience, a well-designed process empowers people to enact changes with confidence.

Ubuntu is built by all of us, in many countries and across all timezones. Concise, well-defined, lightweight processes promote autonomy and reduce uncertainty - enabling people to unblock themselves. Ubuntu is no stranger to process: the Main Inclusion Review (MIR), the aforementioned Stable Release Updates (SRU) process, the process for Snap store requests and many more have contributed to the success of Ubuntu, setting clear guardrails for contributors and ensuring we work to common standards.

My goal over the coming months is to work with you, the people behind Ubuntu, to identify which of these processes still serve us, and which need revising to simplify our work while maintaining our dedication to stability. I’ll consolidate the definitions of these processes, make them searchable, peer-reviewable, and more discoverable. Examples of where this has worked well are the Go proposal process, and the Ethereum Improvement Proposal process - both of which make it trivial to create, track and discuss proposals across the breadth of their respective projects.

If you submit an MIR, or work on an SRU, it should be trivial to understand the status of that request, and to communicate with the team executing that process where needed. If you’re interested in joining our community, it should be simple to get a sense of what is changing across the project, and where you might be able to help.

I’d like to tackle these problems and make these processes as transparent as possible.

Modernisation #

The world of computing has evolved dramatically in the last 20 years, and I’m proud that Ubuntu has continually adapted and thrived. In Linux alone there have been huge changes to what is considered “normal” for a Linux machine. Whether it be the introduction of `systemd`, the advent of languages with a focus on memory safety, the huge growth in virtualisation and containerisation technology, or even the introduction of Rust into the Linux kernel itself - the foundations of our distribution must be constantly assessed against the needs of our users.

I was proud to see the announcement last year that the Ubuntu Kernel team committed to shipping the very latest kernels in new versions of Ubuntu, wherever they possibly can. Even if that means shipping a kernel that’s in the release candidate phase, the team will stand by that kernel and continue to support it through the Ubuntu release’s life. While this could appear cavalier at a glance, what it represents is a willingness to rise to the challenge of shipping the very best of open source to our users. I’d like to see more of this. Ubuntu is a flagship Linux distribution and a starting point for many; we must ensure that our users are presented with the very best our community has to offer - even if that means a bit more hustle in the early days of a given release. This is of particular importance for our Long Term Support releases, which are relied upon by governments, financial institutions, educational establishments, nonprofits and many others for years after the initial release date.

We should look deeply at the tools we ship with Ubuntu by default - selecting for tools that have resilience, performance and maintainability at their core. There are countless examples in the open source community of tools being re-engineered, and re-imagined using tools and practices that have only relatively recently become available. Some of my personal favourites include command-line utilities such as eza, bat, and helix, the new ghostty terminal emulator, and more foundational projects such as the uutils rewrite of coreutils in Rust. Each of these projects are at varying levels of maturity, but have demonstrated a vision for a more modern Unix-like experience that emphasises resilience, performance and usability.

Another example of this is our work on TPM-backed full disk encryption, a project which promises encryption of our users’ data with no degradation to their user experience. This feature relies upon cryptographic hardware and techniques that have only recently become available to us, but enable us to deliver the potent combination of security and usability to our users.

Delivering Features #

What I’ve shared so far is a high-level overview, and many of the points under the four themes will take time to implement, with most appearing as a series of gradual improvements. You might be wondering whether we’ll focus on the latest trends and features, or prioritise that bug you reported.

While focusing on the latest trends or a single breakthrough feature can yield short-term progress, embracing these principles will create the space for sustained, impactful innovation.

That said, I’ve also been working on a list of incremental features and improvements that we can deliver in the coming months to enhance the Ubuntu experience. You’ll hear more from me and the team leads regularly as we share updates and progress.

Summary #

I’m incredibly excited to embark on this journey, and consider it a privilege to serve in this role. Together with the Ubuntu community, Canonical engineers, and our partners, we will build an open-source platform that enables the next 20 years of innovation in computing.

If you have ideas for the future of Ubuntu, or something in this post has resonated with you and you want to be involved either as a community member, or perhaps a future employee of Canonical, I’d love to hear from you.

Packaging the Multipass Flutter GUI for NixOS

Thu, 16 Jan 2025 00:00:00 +0000

Introduction #

The very first application I ever packaged in nixpkgs was Multipass. Multipass is, according to the website:

a CLI to launch and manage VMs on Windows, Mac and Linux that simulates a cloud environment with support for cloud-init.

I wrote in some detail about how I use Multipass and LXD on my workstation to create and manage VMs for testing and development. Last year Multipass shipped a new GUI written from the ground up in Flutter. It provides a clean, modern way to launch, manage and interact with VMs. When the GUI first shipped, I briefly attempted to get it to build with Nix, but some combination of my lack of knowledge, and the maturity of the Flutter tooling in nixpkgs at the time meant I never finished it.

As version 1.15.0 was in its release candidate stage, I decided to have another go!

This post will break down the process into steps, but if you’d like the tl;dr, take a look at the pull request.

Housekeeping #

Before getting started on packaging the GUI, I did some housekeeping on the Multipass package. First order of business was to simply bump the version to 1.15.0 and ensure the package still built, which it did.

Last year, RFC 140 was introduced to simplify the directory structure of nixpkgs, introducing a new pkgs/by-name directory which will (eventually) render pkgs/top-level/all-packages.nix useless. This new structure will make finding package definitions much easier, as they’ll all be arranged alphabetically by the first two characters of their name. Using the Multipass example, rather than pkgs/tools/virtualization/multipass/default.nix, the new preferred path would be pkgs/by-name/mu/multipass/package.nix. There was a talk at NixCon 2023 which summarised this nicely if you’d like more information.

As I was going to be carrying out some hefty work on Multipass by introducing the GUI, I took the opportunity to move to the new scheme as part of the changes.

Restructuring #

The Multipass package carries a few patches, and is already reasonably involved, so I decided that the best route forward was to create separate files for multipassd/multipass (the server & command line client binaries, written in C++, and already packaged) and multipass.gui (the new Flutter application). Thus the new structure for the package looks like so:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


.
# Derivation files
├── gui.nix
├── multipassd.nix
├── package.nix
# Lock file for Fluter app
├── pubspec.lock.json
# Update script for daemon/cli & GUI
├── update.sh
# Patches
├── cmake_no_fetch.patch
├── cmake_warning.patch
├── lxd_socket_path.patch
├── test_unreachable_call.patch
└── vcpkg_no_install.patch

Using `symlinkJoin` #

This package now takes a more unusual shape. Because the GUI is essentially a entirely separate application, written with a different toolkit in a different language, but I wanted the two to be installed as part of the overall multipass package, I decided to use two completely separate derivations and use symlinkJoin to bundle the two together. According to the manual, symlinkJoin:

can be used to put many derivations into the same directory structure. It works by creating a new derivation and adding symlinks to each of the paths listed. It expects two arguments, name, and paths. name (or alternatively pname and version) is the name used in the Nix store path for the created derivation. paths is a list of paths that will be symlinked. These paths can be to Nix store derivations or any other subdirectory contained within.

This is particularly useful for the Multipass package, because the Multipass GUI dynamically-links against libraries that are built as part of the multipassd derivation, and linking the two derivations in this way ensures that all those files are located correctly at runtime without any extra plumbing. This allowed me to keep the top-level package.nix relatively simple (annotated below):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55


{
# ... snip ...
}:
let
 name = "multipass";
 version = "1.15.0";

 # Fetch the source just once, to be used in both the multipassd and GUI derivations
 multipass_src = fetchFromGitHub {
 owner = "canonical";
 repo = "multipass";
 rev = "refs/tags/v${version}";
 hash = "sha256-RoOCh1winDs7BZwyduZziHj6EMe3sGMYonkA757UrIU=";
 fetchSubmodules = true;
 };

 # Shared metadata between derivations.
 commonMeta = {
 homepage = "https://multipass.run";
 license = lib.licenses.gpl3Plus;
 maintainers = with lib.maintainers; [ jnsgruk ];
 platforms = [ "x86_64-linux" ];
 };

 # Build the multipass server & client binaries from the shared source
 # and common metadata.
 multipassd = callPackage ./multipassd.nix {
 inherit commonMeta multipass_src version;
 };

 # Build the multipass GUI using shared source and common metadata.
 multipass-gui = callPackage ./gui.nix {
 inherit commonMeta multipass_src multipassd version;
 };
in
# Create the top-level `multipass` package using `symlinkJoin`
symlinkJoin {
 inherit version;
 pname = name;

 # Join both of the newly created derivations
 paths = [ multipassd multipass-gui ];

 # Ensure tests are run, and define a (new) update script.
 passthru = {
 tests = lib.optionalAttrs stdenv.hostPlatform.isLinux {
 inherit (nixosTests) multipass;
 };
 updateScript = ./update.sh;
 };

 meta = commonMeta // {
 description = "Ubuntu VMs on demand for any workstation";
 };
}

Building the GUI #

Normally the server, client and GUI binaries are all built as part of a single invocation of cmake. We can’t do that here, so the early code I put into the multipass package when I failed to package the GUI before still stood - essentially stopping cmake from trying to build the Flutter GUI:

1
2
3
4


{
 # We'll build the flutter application separately using buildFlutterApplication
 cmakeFlags = [ "-DMULTIPASS_ENABLE_FLUTTER_GUI=false" ];
}

Next, I created gui.nix which uses the pkgs.buildFlutterApplication helper. The derivation is annotated below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67


{
 # Passed in from the package.nix shown above
 commonMeta,
 multipass_src,
 multipassd,
 version,
 # ... snip ...
}:
flutter327.buildFlutterApplication {
 inherit version;
 pname = "multipass-gui";
 src = multipass_src;

 # Ensure we build in the correct directory for the GUI code.
 sourceRoot = "${multipass_src.name}/src/client/gui";

 # To make the builds as deterministic as possible, nixpkgs reads the pubspec.lock
 # file for the Flutter application and downloads/caches the correct versions of
 # flutter packages to be used in the build. It does this with `pub2nix`.
 #
 # Here I use a JSON representation of the pubspec.lock file committed into nixpkgs.
 pubspecLock = lib.importJSON ./pubspec.lock.json;

 # Some of the Flutter dependencies used are pulled from Github (branches) directly, so this
 # ensures that the versions/commits are pinned for future builds.
 gitHashes = {
 dartssh2 = "sha256-2pypKwurziwGLZYuGaxlS2lzN3UvJp3bRTvvYYxEqRI=";
 hotkey_manager_linux = "sha256-aO0h94YZvgV/ggVupNw8GjyZsnXrq3qTHRDtuhNv3oI=";
 system_info2 = "sha256-fly7E2vG+bQ/+QGzXk+DYba73RZccltdW2LpZGDKX60=";
 tray_menu = "sha256-riiAiBEms+9ARog8i+MR1fto1Yqx+gwbBWyNbNq6VTM=";
 window_size = "sha256-71PqQzf+qY23hTJvcm0Oye8tng3Asr42E2vfF1nBmVA=";
 xterm = "sha256-h8vIonTPUVnNqZPk/A4ZV7EYCMyM0rrErL9ZOMe4ZBE=";
 };

 # ... snip ...

 # The Multipass GUI relies on protobuf for API communication; ensure the
 # Dart protobuf definitions are compiled before we build.
 preBuild = ''
 mkdir -p lib/generated

 # Generate the Dart gRPC code for the Multipass GUI.
 protoc \
 --plugin=protoc-gen-dart=${lib.getExe protoc-gen-dart} \
 --dart_out=grpc:lib/generated \
 -I ../../rpc \
 ../../rpc/multipass.proto \
 google/protobuf/timestamp.proto
 '';

 # Libraries built by multipassd are linked dynamically at runtime; multipassd must
 # be installed for the GUI to function.
 runtimeDependencies = [ multipassd ];

 # Housekeeping to ensure the icons/desktop files are correctly installed.
 postFixup = ''
 mv $out/bin/multipass_gui $out/bin/multipass.gui

 install -Dm444 $out/app/multipass-gui/data/flutter_assets/assets/icon.png \
 $out/share/icons/hicolor/256x256/apps/multipass.gui.png

 cp $out/share/applications/multipass.gui.autostart.desktop \
 $out/share/applications/multipass.gui.desktop
 '';

 # ... snip ..
}

Update Script #

That was enough to ensure that the GUI worked correctly. My concern now lied with the maintenance of the package. The process for updating the derivation now involved:

Bumping the version string
Updating the source hashes
Updating the pubspec.lock.json file
Updating the gitHashes of the Flutter dependencies

This seemed a little convoluted and error prone to handle manually each time, so I decided to implement an update script for the package to take care of future version bumps. Thankfully nixpkgs has good provision for this, allowing package maintainers to specify passthru.updateScript to a derivation, which I did here.

I won’t post the full update script here, as it’s lots of unsightly bash! Essentially the script takes care of determining the latest version released on Github by using curl, then updates the pubspec.lock.json using a combination of curl/yj, then finally works through updating the source hash for the Multipass repo, gRPC fork it uses, and each of the git hashes for the Flutter dependencies.

This can be invoked from within the nixpkgs repo with:

1

nix-shell maintainers/scripts/update.nix --argstr package multipass

Summary #

Overall, this turned out cleaner than I expected, though it took me longer to figure out than this post might suggest. My hope is that this will a useful reference to anybody trying to package Flutter applications with Nix, and if you’re a user of the multipass package, check out the shiny new GUI that’s available to you! The GUI is available from 1.15.0 onwards, which at the time of writing means you’ll need to install the package from nixos-unstable or similar (sadly I didn’t land the change in time for NixOS 24.11).

In my opinion, the Multipass team have done a lovely job on the GUI - and for many users this will provide a much improved onboarding experience. Nice work 😎

Experimenting with Rust, Nix, K6 and Parca

Sun, 01 Dec 2024 00:00:00 +0000

Introduction #

Over the past couple of weeks, I’ve been teaching myself Rust. I don’t have a pressing need to write much Rust right now, but I’m intrigued by the promises of memory safety, and have been increasingly impressed at the quality of some of the software that the community produces. I also think that the concepts popularised by Rust, such as the borrow checker, will stick around in computing for many years to come and I’d like to have more hands-on experience with that.

The Rust language also encourages the ideas of safety and soundness - sound code is (approximately) code that can’t cause memory corruption or exhibit undefined behaviour. You can read more in this excellent post Safety and Soundness in Rust.

This blog post started out life as a post about Rust and my experience learning it, but I got interested in the performance of the server implementation I came up with and the post evolved into a post more about profiling and load testing!

Learning Rust #

It’s been a little while since I dug into a new programming language, so I thought I’d mention how I went about it. In general I’m someone who learns best “by doing”. I normally try to read through some of the basic concepts, then jump to a project that will enable me to exercise them quite quickly. Fortunately, Rust has an excellent official guide in the form of the Rust Book, which covers everything from the obligatory Hello, World!, to concurrency, memory safety, publishing packages on crates.io and more. I thoroughly recommend it.

I’ve also picked up a copy of Rust for Rustaceans which was recommended by a couple of different colleagues - I intend to work through this next.

So, after working through the Rust Book over the course of about a week in my spare time, I needed a project!

Rewriting `gosherve` in Rust #

In my first post on this blog I talked about a small Go project I wrote several years ago named gosherve. This was one of my first Go projects - a simple web server that can serve some static assets, and a set of short-links/redirects which are specified in a Github Gist. It’s been happily running my website for several years, and it felt like a small, but ambitious enough project for my first adventure into Rust - particuarly as over the years gosherve has grown Prometheus metrics and the ability to serve assets from an embedded filesystem.

As I anticipated, naming the new project was the hardest part. I landed on servy, at least for now. I’m reasonably happy with the code - at the time of writing:

It can serve redirects
It can serve embedded web assets
It provides a metrics server on a separate port
It has reasonable unit/integration test coverage

One of the things I love about Go is the built-in ability to embed static assets into a binary through the //go:embed directive, which gives you a pointer to an embedded filesystem. I was able to achieve a similar effect in Rust with axum-embed, which in turn builds upon rust-embed.

I used Nix to create different variants of the build (i.e. a “vanilla” build, and one that serves my website by creating a derivation just for the web content, and another for the jnsgruk binary).

You can run a copy of this website (frozen in time just before I wrote this post!) by running nix run github:jnsgruk/servy#jnsgruk, or build a container for it by running nix build github:jnsgruk/servy#jnsgruk-container.

I was expecting the Rust binary to be quite a lot smaller. I’m not sure why. The old (Go) binary weighs in at 68MB, where the new Rust binary comes in at 67MB.

So I was right! I guess?! The result here is relatively uninteresting - a good chunk of both binaries is just the static assets (images!) that make up this site. At the time of writing the jnsgruk-content derivation evaluates at around 57MB - meaning there is 10MB and 9MB respectively for Go and Rust added by the actual server code.

Basic Load Testing with `k6` #

I’ve been looking for an excuse to play with k6 for a while. According to the documentation:

Grafana k6 is an open-source, developer-friendly, and extensible load testing tool. k6 allows you to prevent performance issues and proactively improve reliability.

In very simple terms, to use k6 you define a script (in Javascript) that outlines a set of requests to make, their success criteria and (optionally) the strategy for ramping up load on the server. It has many more features than I used for this project, but I was impressed with how simple it was to get started. I began by running the following (grabbing k6 from nixpkgs):

1
2
3
4
5


# Start a shell with k6 available
❯ nix shell github:NixOS/nixpkgs/nixos-unstable#k6
# Init a new k6 script
❯ k6 new
Initialized a new k6 test script in script.js. You can now execute it by running `k6 run script.js`.

Now I just needed to work out what I wanted to test! I wanted a relatively simple test that requested a mix of web assets and redirects for a sustained period, to see how much throughput I could achieve with each of the server implementations. The template came with some sensible setup for the number of VUs (virtual users) and duration of the test:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


import { check } from "k6"
import http from "k6/http"

export const options = {
 // A number specifying the number of VUs to run concurrently.
 vus: 10,
 // A string specifying the total duration of the test run.
 duration: "30s",
}
// ...

VUs are an emulation of a user interacting with your service; each of them is an agent which will execute the test script. Each time the script is executed (by a VU), that’s known as an “iteration”.

I wanted to test a variety of requests, so I first generated a list of valid URLs for my server. I used cariddi which is a web crawler written in Go, combined with jq to create a list of valid paths in a JSON file:

1
2
3
4
5
6
7


# Start a shell with k6 available
❯ nix shell github:NixOS/nixpkgs/nixos-unstable#cariddi
# Run cariddi and generate the paths.json
❯ echo http://localhost:8080 \
 | cariddi -- -plain \
 | cut -d"/" -f4- \
 | jq -r -nR '[inputs | select(length>0)]' > paths.json

Similarly I generated a list of valid redirects:

1
2
3


❯ curl -s "https://gist.githubusercontent.com/jnsgruk/b590f114af1b041eeeab3e7f6e9851b7/raw" \
 | cut -d" " -f1 \
 | jq -r -nR '[inputs | select(length>0)]' > redirects.json

Now in my k6 script, I was able to read those files to create a list of URLs to batch GET during the load test:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


// Allow the host to be overridden by the 'K6_HOST' env var.
const host = __ENV.K6_HOST || "http://localhost:8080"
// Read the list of paths/redirects.
const assets = JSON.parse(open(`./paths.json`))
const redirects = JSON.parse(open(`./redirects.json`))

export default function () {
 // Batch requests to redirects and static assets.
 const responses = http.batch(
 [...assets, ...redirects].map((p) => {
 return ["GET", `${host}/${p}`, null, { redirects: 0 }]
 })
 )

 // For each response, ensure we get a 200/301/308/404
 // response - we shouldn't see anything else.
 responses.forEach((r) => {
 check(r, {
 status: (res) => {
 return [200, 301, 308, 404].includes(res.status)
 },
 })
 })
}

I’m using a pretty naive check function here, though they can be expanded to check for other conditions in the response body, the size of the response, etc.

You can see my final test script on Github.

Finally, I ran k6 and let it rip! This initial run was against servy, running on my workstation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40


❯ k6 run script.js

 /\  Grafana /‾‾/
 /\  / \  |\  __ / /
 / \/ \  | |/ / / ‾‾\
 / \  | ( | (‾) |
 / __________ \  |_|\_\  \_____/

 execution: local
 script: script.js
 output: -

 scenarios: (100.00%) 1 scenario, 10 max VUs, 1m0s max duration (incl. graceful stop):
 * default: 10 looping VUs for 30s (gracefulStop: 30s)

INFO[0030] [k6-reporter v2.3.0] Generating HTML summary report source=console

 ✓ status

 checks.........................: 100.00% ✓ 1331889 ✗ 0
 data_received..................: 148 GB 4.9 GB/s
 data_sent......................: 163 MB 5.4 MB/s
 http_req_blocked...............: avg=2.41µs min=390ns med=1.72µs max=17.3ms p(90)=3.6µs p(95)=4.91µs
 http_req_connecting............: avg=4ns min=0s med=0s max=462.21µs p(90)=0s p(95)=0s
 http_req_duration..............: avg=501.25µs min=42.08µs med=139.77µs max=45.41ms p(90)=291.59µs p(95)=419.65µs
 { expected_response:true }...: avg=501.25µs min=42.08µs med=139.77µs max=45.41ms p(90)=291.59µs p(95)=419.65µs
 http_req_failed................: 0.00% ✓ 0 ✗ 1331889
 http_req_receiving.............: avg=379.39µs min=4.13µs med=29.07µs max=45.23ms p(90)=108.31µs p(95)=177.05µs
 http_req_sending...............: avg=4.45µs min=960ns med=3.48µs max=3.36ms p(90)=6.83µs p(95)=8.79µs
 http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
 http_req_waiting...............: avg=117.4µs min=33.65µs med=98.19µs max=23.03ms p(90)=176.34µs p(95)=225.09µs
 http_reqs......................: 1331889 44333.219428/s
 iteration_duration.............: avg=38.08ms min=4.92ms med=45.46ms max=80.35ms p(90)=47.68ms p(95)=48.53ms
 iterations.....................: 7881 262.326742/s
 vus............................: 10 min=10 max=10
 vus_max........................: 10 min=10 max=10


running (0m30.0s), 00/10 VUs, 7881 complete and 0 interrupted iterations
default ✓ [======================================] 10 VUs 30s

Nice! Between the 10 VUs we configured, k6 managed nearly 8000 iterations, and my new server responded with a total of 148GB of data!

Nix-ified Load Testing #

Now I’d figured out the basics of k6, I wanted to create some infrastructure that would allow me to run load tests in consistent environments against both old and new implementations of my server.

A couple of ideas came to mind here - the first of which was the NixOS integration test driver. I absolutely love this feature of Nix, and the scripts for interacting with the driver are nice and simple to maintain. One slight irritation in this case is that the test machines don’t have access to the internet - which is where the redirects map is fetched from in my precompiled server binaries. It’s certainly possible to get clever with a fake redirects server and some DNS shennanigans in the test machines, but I opted instead to build simple nixosConfigurations which could be started as VMs, similar to the approach taken in a previous post.

I wanted to automate:

The creation of a VM with both the Go and Rust versions of my server
The generation of the paths.json and redirects.json files I created above
Running the k6 load test
Fetching the results from the load test

Defining Test VMs #

I created a new repository with a flake.nix which took nixpkgs, my jnsgr.uk repo and my servy repo as inputs, pinned to the latest revisions at the time of writing:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


{
 description = "web server benchmarking flake";

 inputs = {
 nixpkgs.url = "github:nixos/nixpkgs/nixpkgs-unstable";
 jnsgruk-go.url = "github:jnsgruk/jnsgr.uk/3240a0104c01ae672a6f5f7b0529ad08bcbc8af2";
 jnsgruk-rust.url = "github:jnsgruk/servy/71a337317defc779c6c55d486e20d104c5d478f2";
 };

 # ...
}

Next up I needed a virtual machine definition, which I first defined as a nixosConfiguration in the flake’s outputs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


{
 # ... inputs ...
 outputs = { self, nixpkgs, jnsgruk-go, jnsgruk-rust, ... }:
 let
 forAllSystems = nixpkgs.lib.genAttrs [ "x86_64-linux" "aarch64-linux" ];
 in
 {
 # A minimal NixOS virtual machine which used for testing craft applications.
 nixosConfigurations = forAllSystems (system: {
 benchvm = nixpkgs.lib.nixosSystem {
 specialArgs = {
 inherit self system;
 jnsgruk-go = jnsgruk-go.packages.${system}.jnsgruk;
 jnsgruk-rust = jnsgruk-rust.packages.${system}.jnsgruk;
 };
 modules = [ ./vm.nix ];
 };
 });
 };
}

The actual machine configuration lives in vm.nix. This is a standard NixOS configuration, with the addition of some elements that define the specs of the virtual machine used to boot it (cores, memory, disk, etc.). In this case the machine is configured with just a root user, and the password password. This is not an ideal setup from a security standpoint, but these VMs were only run on my workstation (behind NAT) for a short period of time and the plain text password eased the automation I’m about to describe. A more robust approach would have been to put my SSH public keys into the authorized keys definition for the user, but then you folks wouldn’t have been able to play along as easily!

In this particular configuration, the VM is set up like so:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


virtualisation = {
 forwardPorts = [{
 from = "host";
 host.port = 2222;
 guest.port = 22;
 }];
 cores = 4;
 memorySize = 4096;
 diskSize = 10000;
};

With that in place, we can now boot the VM (in the background):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


❯ nix run .#nixosConfigurations.benchvm.config.system.build.vm -- --daemonize --display none
❯ ssh -p 2222 root@localhost
The authenticity of host '[localhost]:2222 ([127.0.0.1]:2222)' can't be established.
ED25519 key fingerprint is SHA256:icnH3EQAzmjdfCkyPWFljQWaVSCaXdP2M+ekKXd0NlY.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[localhost]:2222' (ED25519) to the list of known hosts.
(root@localhost) Password:

[root@benchvm:~]#

Wrapping `k6` #

Now on to automating the test itself. When configuring the VM, I made sure to include a systemd unit for each of the server implementations, so next I wanted to write a simple script that would execute the load test for each implementation and fetch the results.

Before that, I modified the k6 script to not only output a report to stdout, but also to a text file, a JSON file and a rendered HTML file (I didn’t know at the time which I’d prefer, and I wanted to survey the options!). I achieved this by defining the handleSummary function in the k6 script:

1
2
3
4
5
6
7
8


export function handleSummary(data) {
 return {
 "summary.json": JSON.stringify(data),
 "summary.html": htmlReport(data),
 "summary.txt": textSummary(data, { enableColors: false, indent: "" }),
 stdout: "\n" + textSummary(data, { enableColors: true, indent: "" }) + "\n\n",
 }
}

First I scripted the generation of the paths and the execution of the test script, and packaged the combination up as a Nix package for installation into the test machine. This is just a bash script which ensures only the specified implementation is running (with systemd), generates the list of URLs to test and then runs k6 against the server. That script is packaged as a Nix package called k6test which contains both my bash script and the k6 test script; the Nix package is then added to the VM configuration.

Automating Test Execution #

I borrowed the vm-exec script from a past project, renaming it benchvm-exec, and created a new script called benchvm-test:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


#!/usr/bin/env bash
set -euo pipefail
info() { echo -e "\e[92m[$HOSTNAME] $*\e[0m"; }

# Set some SSH options to ignore host key errors and make logging quieter.
# This is a bad idea in general, but here is used to faciliate comms with
# a brand new VM each time.
SSH_OPTS=(-P 2222 -o "UserKnownHostsFile=/dev/null" -o "StrictHostKeyChecking=no")

info "Running k6test against $1"
benchvm-exec k6test "$1"

info "Collecting results files"
sshpass -ppassword scp "${SSH_OPTS[@]}" root@localhost:summary.json "summary-${1}.json"
sshpass -ppassword scp "${SSH_OPTS[@]}" root@localhost:summary.html "summary-${1}.html"
sshpass -ppassword scp "${SSH_OPTS[@]}" root@localhost:summary.txt "summary-${1}.txt"

info "Results available in summary-${1}.txt, summary-${1}.json and summary-${1}.html"

This was the final piece of the puzzle for now; after building and packaging this script and defining the test VM as a package in the flake, I’d enabled the following workflow:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Start the devShell for the server-bench project
❯ nix develop github:jnsgruk/server-bench
# Build & run the VM
❯ run-benchvm-vm --daemonize --display none
# Start, and load test the gosherve based server
❯ benchvm-test jnsgruk-go
# Start, and load test the Rust based server
❯ benchvm-test jnsgruk-rust
# All done, power down the VM
❯ benchvm-exec poweroff

Excellent! Now on to some actual testing!

Initial Load Test Results #

Having seen the results for servy earlier, I started up the existing gosherve based server and ran the same test, only to discover quite a delta in the results. Where servy managed 4.9 GB/s throughout the test, gosherve only achieved 697 MB/s (sending 21GB in total). The full results are in the details box below, but overall it managed much lower performance across the board on the measurements that k6 makes.

Initial gosherve load-test results

checks.........................: 100.00% ✓ 189449 ✗ 0
data_received..................: 21 GB 697 MB/s
data_sent......................: 26 MB 848 kB/s
http_req_blocked...............: avg=2.02µs min=400ns med=1.42µs max=19.76ms p(90)=2.29µs p(95)=2.77µs
http_req_connecting............: avg=203ns min=0s med=0s max=19.52ms p(90)=0s p(95)=0s
http_req_duration..............: avg=9.3ms min=29.49µs med=9.36ms max=152.39ms p(90)=13.16ms p(95)=14.76ms
{ expected_response:true }...: avg=9.3ms min=29.49µs med=9.36ms max=152.39ms p(90)=13.16ms p(95)=14.76ms
http_req_failed................: 0.00% ✓ 0 ✗ 189449
http_req_receiving.............: avg=106.56µs min=4.88µs med=48.7µs max=6.01ms p(90)=215.06µs p(95)=381.38µs
http_req_sending...............: avg=4.81µs min=950ns med=3.97µs max=19.71ms p(90)=5.93µs p(95)=6.98µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=9.19ms min=22.85µs med=9.26ms max=152.2ms p(90)=12.96ms p(95)=14.51ms
http_reqs......................: 189449 6281.256789/s
iteration_duration.............: avg=268.14ms min=176.88ms med=250.61ms max=433.23ms p(90)=334.14ms p(95)=342.1ms
iterations.....................: 1121 37.1672/s
vus............................: 10 min=10 max=10
vus_max........................: 10 min=10 max=10

I was quite surprised by this: my expectation was that the Go HTTP server would outperform what I’d put together in Rust. I decided to look a little deeper and see if I could figure out why, or at least why the delta was so big.

Profiling `gosherve` with Parca #

For last couple of years, I’ve been playing with Parca, which is a continuous profiling tool written in Go by the folks at Polar Signals. I wrote about it back in 2022 on Charmhub, demonstrating how Parca could be used for profiling applications deployed with Juju.

Instrumenting `gosherve` #

There are two components to the open source Parca offering - the server backend, and a zero-instrumentation, eBPF-based agent which can be used for on-CPU profiling of any workload. Because gosherve is Go based, I didn’t need the agent so long as I enabled the pprof endpoint in my server, which is trivial for almost any Go application. I took main.go and applied the following changes to ensure that I could hit the pprof endpoints on port 6060:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


diff --git a/main.go b/main.go
index 964ceb2..374c440 100644
--- a/main.go
+++ b/main.go
@@ -10,6 +10,9 @@ import (
 "log/slog"
 "os"

+ "net/http"
+ _ "net/http/pprof"
+
 "github.com/jnsgruk/gosherve/pkg/logging"
 "github.com/jnsgruk/gosherve/pkg/server"
 )
@@ -29,6 +32,11 @@ func main() {
 flag.Parse()
 logging.SetupLogger(*logLevel)

+ go func() {
+ http.ListenAndServe(":6060", nil)
+ slog.Info("pprof server started on port 6060")
+ }()
+
 // Create an fs.FS from the embedded filesystem
 fsys, err := fs.Sub(publicFS, "public")
 if err != nil {

Next I fired up Parca (having first packaged it for NixOS 😉) and configured it to scrape the new jnsgruk binary while I ran the same test against it, which resulted in the following profile (which you can explore yourself on pprof.me):

Looking at this, we can see that around a third of the time was spent in garbage collection. This seemed high; it was more prominent on this run than others but it was persistently a high portion of the CPU time. Something to loop back to! However, inside the route handler itself there are two things that stood out to me:

Initial optimisations #

In the red box, you can see about a third of the time spent in the route handler was in creating SHA1 sums, which is happening here to calculate the ETag for content before serving it. This could probably be optimised - either by changing how the ETag is calculated (i.e. not hashing the whole file contents), or by caching ETags in memory, particularly given that in the case of my website, gosherve only serves assets which are embedded (read only) in the binary, so the ETag will never change for a given file for a particular compiled binary.

In the blue box, we can see that another third of the time spent is on refreshing the redirects. This seemed strange; the redirects don’t change throughout the life of the test, and we’re only requesting known redirects from our redirects.json file. This seemed much more likely to be a culprit in the poor load test results, because unlike calculating a SHA1 sum (which is almost entirely CPU-bound), this function is making an outbound (TLS) connection to Github and potentially blocking on IO before parsing the response, rechecking the redirects map for a match, etc.

As I looked at the code, I noticed an obvious (and somewhat embarrassing!) mistake I’d made when implementing gosherve. Because the server tries to parse redirects before looking for files with a matching path, every request for a file causes the redirects map to be updated from its upstream source 🤦. Looking back at the new servy implementation, I hadn’t made the same mistake - perhaps this was the reason the Rust version was so much faster?

I made a small change to gosherve and re-ran the test, which resulted in this profile and quite a substantial bump in request performance!

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


checks.........................: 100.00% ✓ 913783 ✗ 0
data_received..................: 101 GB 3.4 GB/s
data_sent......................: 123 MB 4.1 MB/s
http_req_blocked...............: avg=3.21µs min=390ns med=2.12µs max=14.12ms p(90)=4.19µs p(95)=5.7µs
http_req_connecting............: avg=70ns min=0s med=0s max=14.09ms p(90)=0s p(95)=0s
http_req_duration..............: avg=1.8ms min=42.12µs med=899.13µs max=31.31ms p(90)=4.86ms p(95)=6.45ms
 { expected_response:true }...: avg=1.8ms min=42.12µs med=899.13µs max=31.31ms p(90)=4.86ms p(95)=6.45ms
http_req_failed................: 0.00% ✓ 0 ✗ 913783
http_req_receiving.............: avg=261.88µs min=4.85µs med=63.59µs max=17.8ms p(90)=620.15µs p(95)=1.31ms
http_req_sending...............: avg=8.21µs min=1.05µs med=5.21µs max=8.73ms p(90)=9.87µs p(95)=13.07µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=1.53ms min=30.81µs med=656.71µs max=30.76ms p(90)=4.38ms p(95)=5.91ms
http_reqs......................: 913783 30435.133157/s
iteration_duration.............: avg=55.5ms min=28.86ms med=55.18ms max=77.07ms p(90)=61.02ms p(95)=62.98ms
iterations.....................: 5407 180.089545/s
vus............................: 10 min=10 max=10
vus_max........................: 10 min=10 max=10

The overall throughput increased by five times, with a decrease in latency across the board. Still around a third less overall throughput than the Rust server, but a huge improvement nonetheless.

I subsequently experimented with (naively) removing the ETag calculation to see whether or not it was worth implementing some caching - but it actually resulted in very little difference in throughput and CPU utilisation. Besides, while the calculation takes some CPU time, in a real-life deployment it reduces the overall load on the server by ensuring that http.ServeContent only sends actual content when there is a new version, relying more heavily on the user’s browser cache. My k6 tests (deliberately) weren’t sending the right headers (such as If-None-Match) to benefit from the behaviour one might expect from a browser, and it was just requesting the same files over and over again with the equivalent of a cold/empty cache.

Reducing Allocations #

Now back to all those allocations, and the time spent in garbage collection as a result… gosherve was spending about a third of it’s time in the io.ReadFile function, and given the amount of time spent in garbage collection, any optimisation made to the number of allocations on the heap would likely yield another big performance increase.

The problem lied in the routeHandler, where I was reading the entire contents of each file that was being served into memory. fs.ReadFile makes an allocation the size of the file it’s reading, meaning the contents of the file end up on the heap. In a situation like our load test - this means the entire contents of my website ended up on the heap, and the Go runtime was busily garbage collecting to clean up.

I looked around at alternative ways to implement the same functionality in a more efficient manner. In Go 1.22, http.ServeFileFS was introduced as a counterpart to http.ServeFile. http.ServeFileFS operates on an fs.FS rather than the host filesystem. Under the hood, rather than reading the whole file into memory ServeFileFS is using io.Copy and thus io.copyBuffer which allocates a single buffer that fits on the stack - fewer allocations means less memory usage, and less time spent occupying cores with garbage collection!

The change to gosherve was pretty simple - you can see a slightly shortened diff below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


diff --git a/pkg/server/route_handler.go b/pkg/server/route_handler.go
index 2417e45..e0c9d68 100644
--- a/pkg/server/route_handler.go
+++ b/pkg/server/route_handler.go
@@ -1,7 +1,6 @@
 // snip!

- // Try reading the file and return early if that fails
- b, err := fs.ReadFile(*s.webroot, filepath)
- if err != nil {
- return false
- }
-
 w.Header().Set("Cache-Control", "public, max-age=31536000, must-revalidate")
- w.Header().Set("ETag", fmt.Sprintf(`"%d-%x"`, len(b), sha1.Sum(b)))
+ w.Header().Set("ETag", fmt.Sprintf(`"%s-%d-%x"`, fi.Name(), fi.Size(), sha1.Sum([]byte(fi.ModTime().String()))))

- http.ServeContent(w, r, filepath, time.Now(), bytes.NewReader(b))
+ http.ServeFileFS(w, r, *s.webroot, filepath)
 s.metrics.responseStatus.WithLabelValues(strconv.Itoa(http.StatusOK)).Inc()
 l.Info("served file", slog.Group("response", "status_code", http.StatusOK, "file", filepath))

This change also necessitated a change to ETag calculation, which now only hashes the filename, size and modified time. There is a slight oddity in that files from an embedded filesystem have their timestamps all fixed (to avoid changes in static files messing with build reproducibility), but since the files cannot change during a binary’s lifetime, the ETag calculated from the modified timestamp is still just as valid.

I quickly made a new build of jnsgruk using a locally referenced copy of gosherve and ran k6/parca to check the outcome. You can explore the profile yourself on pprof.me:

Here you can clearly see that the runtime is spending much less time in garbage collection, and the call to io.Copy/io.copyBuffer without the calls to fs.ReadFile. It’s now also much harder to spot the ETag generation - likely because we’re now only hashing a small string, rather than the whole file contents.

The k6 results came back as follows 🚀:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


checks.........................: 100.00% ✓ 1525300 ✗ 0
data_received..................: 170 GB 5.7 GB/s
data_sent......................: 207 MB 6.9 MB/s
http_req_blocked...............: avg=4µs min=410ns med=2.36µs max=18.47ms p(90)=5.02µs p(95)=6.77µs
http_req_connecting............: avg=40ns min=0s med=0s max=17.98ms p(90)=0s p(95)=0s
http_req_duration..............: avg=999.6µs min=42.17µs med=645.03µs max=24.22ms p(90)=2.34ms p(95)=3.09ms
 { expected_response:true }...: avg=999.6µs min=42.17µs med=645.03µs max=24.22ms p(90)=2.34ms p(95)=3.09ms
http_req_failed................: 0.00% ✓ 0 ✗ 1525300
http_req_receiving.............: avg=245.43µs min=4.64µs med=75.29µs max=23.48ms p(90)=626.08µs p(95)=1.18ms
http_req_sending...............: avg=9.69µs min=1µs med=5.49µs max=19.54ms p(90)=10.86µs p(95)=14.46µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=744.47µs min=28.09µs med=395.81µs max=20.62ms p(90)=1.89ms p(95)=2.58ms
http_reqs......................: 1525300 50823.211595/s
iteration_duration.............: avg=34.41ms min=11.91ms med=34.44ms max=67.75ms p(90)=37.6ms p(95)=38.61ms
iterations.....................: 8716 290.418352/s
vus............................: 10 min=10 max=10
vus_max........................: 10 min=10 max=10

The Go server is now in the lead! There isn’t a huge margin here between the Rust server and the Go server, but this is about the sort of difference I was expecting based on the reading I’d done before doing the testing.

Profiling `servy`? #

I did profile servy using parca-agent, but I’ve yet to look at the results in detail. The profile is a lot more complex (see below), partially as a result of how much work is going on under the hood when standing up an axum server with hyper, tower and tokio. I’m going to spend some more time with this profile over the coming weeks, but you can explore it for yourself on pprof.me.

If you do play with the profile on pprof.me, try filtering by function name and entering servy, and you’ll see the parts of the code I wrote highlighted on the profile to explore:

By the time you read this, the parca-agent should be upstream into nixpkgs, but in the mean time you can pull the package I included in the server-bench repo and use that:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Start the devShell for the server-bench project
❯ nix develop github:jnsgruk/server-bench
# Start the agent
❯ sudo parca-agent -- \
 --remote-store-address="grpc.polarsignals.com:443" \
 --remote-store-bearer-token="$TOKEN" \
 --node="$HOSTNAME"

# Or, on Ubuntu (see polarsignals.com/docs/setup-collection-snaps)
❯ sudo snap install --classic parca-agent

In my Nix example above, I’m sending profiles to Polar Signals Cloud, but you can also set the remote-store-address to the address of a locally running parca instance no problem.

Depending on the language you’re profiling, you’ll either need to include debug info in the binary you’re profiling (which is what I did), or upload your source/debug info manually. There are more instructions for that in the docs.

Final Results & Comparisons #

I wanted to gather some test results from all three targets across a number of different machine configurations; my website runs on a teeny-tiny Fly.io instance with 1 vCPU and 256MB RAM, so I was interested to see how performance changed on machines with varying configurations. I’ve included a summary in the table below which highlights just the data_received measurement from each test, and you can see the full reports from k6 by expanding the details box:

Expand for full `k6` reports

1 vCPU/256MB RAM/1 VUs - gosherve (old)

checks.........................: 100.00% ✓ 23322 ✗ 0
data_received..................: 2.6 GB 86 MB/s
data_sent......................: 3.1 MB 105 kB/s
http_req_blocked...............: avg=2.41µs min=400ns med=1.28µs max=1.04ms p(90)=1.87µs p(95)=2.27µs
http_req_connecting............: avg=50ns min=0s med=0s max=508.13µs p(90)=0s p(95)=0s
http_req_duration..............: avg=7.57ms min=49.54µs med=7.71ms max=143.13ms p(90)=9.42ms p(95)=10.65ms
{ expected_response:true }...: avg=7.57ms min=49.54µs med=7.71ms max=143.13ms p(90)=9.42ms p(95)=10.65ms
http_req_failed................: 0.00% ✓ 0 ✗ 23322
http_req_receiving.............: avg=98.03µs min=4.46µs med=57.25µs max=7.69ms p(90)=177.19µs p(95)=250.6µs
http_req_sending...............: avg=5.7µs min=970ns med=3.1µs max=480.07µs p(90)=4.79µs p(95)=9.26µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=7.47ms min=37.26µs med=7.62ms max=142.9ms p(90)=9.24ms p(95)=10.33ms
http_reqs......................: 23322 776.881114/s
iteration_duration.............: avg=217.52ms min=187.06ms med=198.66ms max=359.52ms p(90)=276.89ms p(95)=280.19ms
iterations.....................: 138 4.59693/s
vus............................: 1 min=1 max=1
vus_max........................: 1 min=1 max=1

1 vCPU/256MB RAM/1 VUs - gosherve

checks.........................: 100.00% ✓ 158691 ✗ 0
data_received..................: 18 GB 587 MB/s
data_sent......................: 21 MB 714 kB/s
http_req_blocked...............: avg=1.89µs min=379ns med=1.08µs max=1.15ms p(90)=1.62µs p(95)=1.98µs
http_req_connecting............: avg=11ns min=0s med=0s max=655.05µs p(90)=0s p(95)=0s
http_req_duration..............: avg=1.04ms min=48.25µs med=537.57µs max=31.33ms p(90)=2.63ms p(95)=3.66ms
{ expected_response:true }...: avg=1.04ms min=48.25µs med=537.57µs max=31.33ms p(90)=2.63ms p(95)=3.66ms
http_req_failed................: 0.00% ✓ 0 ✗ 158691
http_req_receiving.............: avg=91.12µs min=3.66µs med=20.8µs max=23.05ms p(90)=156.29µs p(95)=287.17µs
http_req_sending...............: avg=6.47µs min=900ns med=2.57µs max=2.22ms p(90)=4.06µs p(95)=6.21µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=949.02µs min=35.88µs med=466.14µs max=31.3ms p(90)=2.46ms p(95)=3.47ms
http_reqs......................: 158691 5287.057691/s
iteration_duration.............: avg=31.95ms min=21.16ms med=30.39ms max=107.46ms p(90)=36.19ms p(95)=41.18ms
iterations.....................: 939 31.284365/s
vus............................: 1 min=1 max=1
vus_max........................: 1 min=1 max=1

1 vCPU/256MB RAM/1 VUs - servy

checks.........................: 100.00% ✓ 148213 ✗ 0
data_received..................: 16 GB 547 MB/s
data_sent......................: 18 MB 605 kB/s
http_req_blocked...............: avg=1.5µs min=350ns med=971ns max=1.11ms p(90)=1.48µs p(95)=1.82µs
http_req_connecting............: avg=11ns min=0s med=0s max=451.97µs p(90)=0s p(95)=0s
http_req_duration..............: avg=928.79µs min=69.84µs med=671.38µs max=55.59ms p(90)=1.57ms p(95)=2.18ms
{ expected_response:true }...: avg=928.79µs min=69.84µs med=671.38µs max=55.59ms p(90)=1.57ms p(95)=2.18ms
http_req_failed................: 0.00% ✓ 0 ✗ 148213
http_req_receiving.............: avg=136.98µs min=3.76µs med=17.14µs max=46.31ms p(90)=103.7µs p(95)=192.52µs
http_req_sending...............: avg=5.33µs min=890ns med=2.35µs max=2.99ms p(90)=3.69µs p(95)=5.05µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=786.47µs min=55.89µs med=629.47µs max=17.92ms p(90)=1.43ms p(95)=1.96ms
http_reqs......................: 148213 4940.080272/s
iteration_duration.............: avg=34.19ms min=16.57ms med=25.68ms max=109.45ms p(90)=55.08ms p(95)=57.83ms
iterations.....................: 877 29.231244/s
vus............................: 1 min=1 max=1
vus_max........................: 1 min=1 max=1

2 vCPU/4GB RAM/10 VUs - gosherve (old)

checks.........................: 100.00% ✓ 116272 ✗ 0
data_received..................: 13 GB 426 MB/s
data_sent......................: 16 MB 518 kB/s
http_req_blocked...............: avg=2.25µs min=380ns med=1.06µs max=4.3ms p(90)=1.69µs p(95)=2.05µs
http_req_connecting............: avg=475ns min=0s med=0s max=4.27ms p(90)=0s p(95)=0s
http_req_duration..............: avg=15.37ms min=29.8µs med=14.28ms max=145.82ms p(90)=24.79ms p(95)=29.78ms
{ expected_response:true }...: avg=15.37ms min=29.8µs med=14.28ms max=145.82ms p(90)=24.79ms p(95)=29.78ms
http_req_failed................: 0.00% ✓ 0 ✗ 116272
http_req_receiving.............: avg=174.41µs min=4.15µs med=28.84µs max=21.95ms p(90)=241.72µs p(95)=703.97µs
http_req_sending...............: avg=7.22µs min=890ns med=2.98µs max=5.58ms p(90)=5.8µs p(95)=9.15µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=15.19ms min=22.98µs med=14.11ms max=144.5ms p(90)=24.48ms p(95)=29.37ms
http_reqs......................: 116272 3837.442876/s
iteration_duration.............: avg=439.41ms min=323.27ms med=430.04ms max=567.74ms p(90)=505.87ms p(95)=516.96ms
iterations.....................: 688 22.706763/s
vus............................: 10 min=10 max=10
vus_max........................: 10 min=10 max=10

2 vCPU/4GB RAM/10 VUs - gosherve

checks.........................: 100.00% ✓ 362167 ✗ 0
data_received..................: 40 GB 1.3 GB/s
data_sent......................: 49 MB 1.6 MB/s
http_req_blocked...............: avg=2.23µs min=360ns med=770ns max=12.31ms p(90)=1.34µs p(95)=1.66µs
http_req_connecting............: avg=117ns min=0s med=0s max=1.84ms p(90)=0s p(95)=0s
http_req_duration..............: avg=4.7ms min=32.38µs med=3.94ms max=47.81ms p(90)=9.04ms p(95)=11.63ms
{ expected_response:true }...: avg=4.7ms min=32.38µs med=3.94ms max=47.81ms p(90)=9.04ms p(95)=11.63ms
http_req_failed................: 0.00% ✓ 0 ✗ 362167
http_req_receiving.............: avg=196.28µs min=3.38µs med=15.42µs max=25.62ms p(90)=142.41µs p(95)=896.21µs
http_req_sending...............: avg=8.44µs min=890ns med=2.19µs max=18.13ms p(90)=3.63µs p(95)=6.34µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=4.5ms min=25.57µs med=3.77ms max=45.46ms p(90)=8.74ms p(95)=11.13ms
http_reqs......................: 362167 12040.737924/s
iteration_duration.............: avg=140.21ms min=80.54ms med=139.82ms max=221.91ms p(90)=153.25ms p(95)=157.95ms
iterations.....................: 2143 71.24697/s
vus............................: 10 min=10 max=10
vus_max........................: 10 min=10 max=10

2 vCPU/4GB RAM/10 VUs - servy

checks.........................: 100.00% ✓ 464412 ✗ 0
data_received..................: 51 GB 1.7 GB/s
data_sent......................: 57 MB 1.9 MB/s
http_req_blocked...............: avg=1.99µs min=360ns med=720ns max=14.38ms p(90)=1.13µs p(95)=1.42µs
http_req_connecting............: avg=92ns min=0s med=0s max=6.71ms p(90)=0s p(95)=0s
http_req_duration..............: avg=3.67ms min=44.91µs med=3.07ms max=60.17ms p(90)=5.9ms p(95)=7.98ms
{ expected_response:true }...: avg=3.67ms min=44.91µs med=3.07ms max=60.17ms p(90)=5.9ms p(95)=7.98ms
http_req_failed................: 0.00% ✓ 0 ✗ 464412
http_req_receiving.............: avg=206.58µs min=3.41µs med=14.06µs max=56.96ms p(90)=56.97µs p(95)=143.43µs
http_req_sending...............: avg=7.12µs min=870ns med=1.62µs max=14.84ms p(90)=3.01µs p(95)=5.78µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=3.45ms min=35.61µs med=3.01ms max=24.64ms p(90)=5.7ms p(95)=7.54ms
http_reqs......................: 464412 15448.582291/s
iteration_duration.............: avg=109.3ms min=63.59ms med=108.88ms max=161.77ms p(90)=120.71ms p(95)=124.65ms
iterations.....................: 2748 91.41173/s
vus............................: 10 min=10 max=10
vus_max........................: 10 min=10 max=10

16 vCPU/16GB RAM/10 VUs - gosherve (old)

checks.........................: 100.00% ✓ 119990 ✗ 0
data_received..................: 13 GB 439 MB/s
data_sent......................: 16 MB 534 kB/s
http_req_blocked...............: avg=2.05µs min=440ns med=1.55µs max=972.62µs p(90)=2.32µs p(95)=2.74µs
http_req_connecting............: avg=38ns min=0s med=0s max=275.54µs p(90)=0s p(95)=0s
http_req_duration..............: avg=14.82ms min=35.49µs med=15.16ms max=148.26ms p(90)=21.12ms p(95)=23.89ms
{ expected_response:true }...: avg=14.82ms min=35.49µs med=15.16ms max=148.26ms p(90)=21.12ms p(95)=23.89ms
http_req_failed................: 0.00% ✓ 0 ✗ 119990
http_req_receiving.............: avg=104.7µs min=5.07µs med=43.71µs max=8.96ms p(90)=228.22µs p(95)=396.05µs
http_req_sending...............: avg=4.52µs min=940ns med=3.99µs max=944.81µs p(90)=6.28µs p(95)=7.86µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=14.71ms min=23.97µs med=15.05ms max=148.25ms p(90)=20.95ms p(95)=23.7ms
http_reqs......................: 119990 3957.203279/s
iteration_duration.............: avg=426.57ms min=343.33ms med=422.38ms max=526.97ms p(90)=491ms p(95)=502.47ms
iterations.....................: 710 23.415404/s
vus............................: 10 min=10 max=10
vus_max........................: 10 min=10 max=10

16 vCPU/16GB RAM/10 VUs - gosherve

checks.........................: 100.00% ✓ 1297244 ✗ 0
data_received..................: 144 GB 4.8 GB/s
data_sent......................: 175 MB 5.8 MB/s
http_req_blocked...............: avg=2.83µs min=419ns med=1.63µs max=7.97ms p(90)=2.95µs p(95)=3.78µs
http_req_connecting............: avg=2ns min=0s med=0s max=142.69µs p(90)=0s p(95)=0s
http_req_duration..............: avg=1.22ms min=31.96µs med=697.97µs max=36.9ms p(90)=3.16ms p(95)=4.07ms
{ expected_response:true }...: avg=1.22ms min=31.96µs med=697.97µs max=36.9ms p(90)=3.16ms p(95)=4.07ms
http_req_failed................: 0.00% ✓ 0 ✗ 1297244
http_req_receiving.............: avg=234.98µs min=4.21µs med=39.77µs max=35.6ms p(90)=640.71µs p(95)=1.24ms
http_req_sending...............: avg=7.19µs min=930ns med=3.91µs max=30.11ms p(90)=6.84µs p(95)=9.22µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=980.01µs min=19.74µs med=465.75µs max=16.64ms p(90)=2.75ms p(95)=3.68ms
http_reqs......................: 1297244 43221.02723/s
iteration_duration.............: avg=39.08ms min=23.12ms med=38.84ms max=76ms p(90)=43.87ms p(95)=45.71ms
iterations.....................: 7676 255.745723/s
vus............................: 10 min=10 max=10
vus_max........................: 10 min=10 max=10

16 vCPU/16GB RAM/10 VUs - servy

checks.........................: 100.00% ✓ 1407770 ✗ 0
data_received..................: 156 GB 5.2 GB/s
data_sent......................: 172 MB 5.7 MB/s
http_req_blocked...............: avg=2.28µs min=420ns med=1.64µs max=3.96ms p(90)=3µs p(95)=3.94µs
http_req_connecting............: avg=2ns min=0s med=0s max=292.26µs p(90)=0s p(95)=0s
http_req_duration..............: avg=512.52µs min=40.21µs med=177.39µs max=45.65ms p(90)=444.41µs p(95)=623.84µs
{ expected_response:true }...: avg=512.52µs min=40.21µs med=177.39µs max=45.65ms p(90)=444.41µs p(95)=623.84µs
http_req_failed................: 0.00% ✓ 0 ✗ 1407770
http_req_receiving.............: avg=324.43µs min=4.02µs med=26.56µs max=45.31ms p(90)=88.04µs p(95)=147.94µs
http_req_sending...............: avg=4.33µs min=969ns med=3.35µs max=4.11ms p(90)=5.86µs p(95)=7.31µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=183.74µs min=31.45µs med=132.09µs max=8.96ms p(90)=344.35µs p(95)=465.04µs
http_reqs......................: 1407770 46854.747614/s
iteration_duration.............: avg=36.03ms min=5.19ms med=45.31ms max=77.29ms p(90)=48.06ms p(95)=48.95ms
iterations.....................: 8330 277.247027/s
vus............................: 10 min=10 max=10
vus_max........................: 10 min=10 max=10

	`gosherve (old)`	`gosherve`	`servy`
1 vCPU/256MB RAM/1 VUs	86MB/s	587MB/s	547MB/s
2 vCPU/4GB RAM/10 VUs	426MB/s	1.3GB/s	1.7GB/s
16 vCPU/16GB RAM/10 VUs	439MB/s	4.8GB/s	5.2GB/s

Interestingly, it seems that the servy variant actually outperformed gosherve again in these VM-based tests, where it didn’t when run directly on my machine. In any case, the gap is now much smaller between the two, and gosherve still seems to maintain an edge on very small machines.

Note I had to drop the VUs to 1 on the smallest iteration, or I was seeing all three of the variants getting OOM-killed!

I also added a new flake input in the server-bench repo that adds the gosherve based repo again, but pinned to the pre-optimised version.

As such, if you want to try reproducing any of these results and measure old vs. new vs. servy, you just need to do the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Start the devShell for the server-bench project
❯ nix develop github:jnsgruk/server-bench
# Build & run the VM
❯ run-benchvm-vm --daemonize --display none
# Edit the core/memory count as required
❯ vim vm.nix
# Start, and load test a server, choosing your variety
❯ benchvm-test <jnsgruk-go-old|jnsgruk-go|jnsgruk-rust>
# Power down the VM
❯ benchvm-exec poweroff

The results will be gathered in a collection of summary-*.<txt|html|json> files for you to inspect.

Summary #

This was supposed to be a blog about learning Rust, but I’m pleased with where it ended up! This was by no means a particularly in-depth dive into measuring the performance of web servers, focusing almost entirely on the request throughput. There are other factors that could be taken into consideration - such as the resulting memory pressure on the machine, CPU usage, etc. I may come back to this in another post with more comprehensive measurement of both.

Hopefully this post illustrated the power of load testing and profiling tools - even when only used at quite a surface level, and demonstrated how Nix can be used to create robust structures around existing tools to help with such workflows.

I owe a thank you to Frederic from Polar Signals, who gave me a bunch of helpful tips along the way while I was using Parca and profiling gosherve. After my initial improvement in the request handler, Frederic helped me track down and remove the use of fs.ReadFile and further improve the performance of gosherve.

I’m still very early in my Rust development, so if you’ve spotted something horrible in servy, or you’ve got experience with k6 and parca, then reach out and let me know! I’d love to see the creative ways people use these tools.

The performance gap between servy and gosherve is now quite narrow. Close enough, in fact, that they seem to win over one another depending on different conditions in the test environment, and never by a particularly high margin. I haven’t migrated this site over yet, but perhaps I will in the near future. Besides, Fly.io are already kindly limiting the damage that can be done by too many requests for me!

See you next time!

Hot Tub Monitoring with Home Assistant and ESPHome

Tue, 12 Nov 2024 00:00:00 +0000

Introduction #

This year we put a wood-fired hot tub in our back garden. It’s a total indulgence and a real treat. I originally purchased the tub for my wife, but I must admit to having grown quite fond of it myself!

The tub itself is made from Canadian Redwood Cedar. The photo below was taken shortly after it was assembled, so there is still a little bit of leakage which stopped a few days later once the wood had expanded. The 30kW wood burner is the most effective way to heat the tub (which holds around 1700l of water), but there is also a small electric heater in line with the pump & filter which are stashed behind the tub.

There is something wonderfully simple and low-tech about the whole arrangement, but naturally I wanted to get an understanding of the energy usage, and get a reading on the temperature so that I could more accurately set it up for when we wanted to use it. Part of the reasoning for understanding the energy usage was to ensure that the pump and UV filter only fire up when the house is generating enough solar energy to cover it.

A picture paints a thousand words, so you can see what I’m talking about below:

Energy Usage Monitoring #

Starting with the easy part: monitoring the energy usage for each of the components. What you can’t see in the picture above is the following:

An Elecro Vulcan 3kw heater
An evoUV 15w UV clarifier
A Crystal Enterprises CC2513 pump
A Crystal Enterprises CC3030 filter unit

The UV filter, heater and sand filter are all essentially “in line” - when the pump is switched on, water is pulled from the bottom of the tub, through the UV clarifier, sand filter and heater, and then back into the tub (not necessarily in that order!). The heater has a thermostat in it so that it can cut in/out depending on the desired temperature as the water passes through.

In reality, the 3kw heater would take 12-15 hours to heat the tub from scratch, which is why we use the burner that usually takes around 1.5-2 hours. The electric heater is good for maintaining the temperature once the initial heating has been done.

I’ve had quite a lot of success with TP-Link Tapo P110 smart plugs. They’re very cheap, and there is already a competent Home Assistant integration for them. The only downside is that they require a proprietary app for initial setup. After having bought a few of these, I learned about Tasmota, and will probably buy smart plugs compatible with this or similar open source firmwares in the future.

My friends from the Self Hosted Podcast seem to recommend buying pre-flashed smart devices from cloudfree.shop, which I’ll certainly consider next time.

All of the above said - the TP-Link plugs work great for this use-case. I’ve got one for the pump and one for the heater. The pump is configured to run for 2 hours a day during peak daylight hours, and the heater can be toggled as and when I need it. The result, when added to Home Assistant is some neat energy monitoring and the ability to toggle things on and off easily when I need:

In an alternative view that summarises power usage for my home, there is a neat section covering the usage from individual devices (all connected to smart plugs):

As you can see - the heater is incredibly power-hungry, and as a result we hardly use it. In the two hours it ran on the day shown above, it consumed almost twice the energy that my main workstation used all day. We now constrain it’s use to a couple of hours per day in the summer months when solar generation is at its peak - otherwise we just heat the tub with the wood burner as we want.

Temperature Sensor: Initial Solution #

When the tub originally arrived, I looked around for an internet-connected pool thermometer, and was quite surprised at the lack of options. I had assumed this was something relatively common. I ended up ordering an Inkbird IBS-P02R kit.

The kit comprises a small floating thermometer, and a display unit that communicates wirelessly with the thermometer. I picked this particular model because the display unit can be connected to Wi-Fi and then checked on through an app. I had (naively…) assumed that I’d be able to get access through some sort of API, but that turned out not to be the case.

Overall, this solution worked okay. The temperature is reported in 5/10/15 minute intervals according to the configuration, and I had no issues with range/connectivity even though the tub is some distance from my house (and the house is heavily insulated with triple glazing!).

The Inkbird app for iOS leaves a lot to be desired, though. I found the user experience pretty frustrating. The web service was often slow to respond, and it required an annoying number of clicks to get to the information I wanted.

While researching the different ways I could get access to the information, I re-discovered the Inkbird IBS-P01B which is a very similar thermometer to the one I’d bought, but communicates over bluetooth. I’d initially ruled this out since my server is well out of bluetooth range, but I’d also been looking for an excuse to play with an ESP32-based microcontroller…

Temperature Sensor: Homemade Solution #

After a little bit of research I ended up ordering the following:

1x Inkbird IBS-P01B thermometer
1x ESP32-WROOM-32U development board

The former was ordered from AliExpress, and the latter from Amazon. I chose the ESP32-WROOM-32U specifically because it has an external antenna. Given the eventual placement of the ESP32 and the location of the hot tub, I wanted to maximise the chances of establishing a good bluetooth connection.

This was to be my first foray into ESP32/Arduino type development. I’d read (and heard…) lots about ESPHome, and given my goal was to integrate with Home Assistant, this felt like the right route.

Connecting the ESP32 #

The first step was to flash ESPHome onto the device. The development board I bought comes with a CP2012 USB to UART bridge on the board, so connecting the device to my machine was as simple as plugging it in with a MicroUSB cable.

Before the device can be flashed, it needs to be put into programming mode. In my case, the board has a handy BOOT button. The process for entering programming mode was therefore (starting with the device unplugged and powered down):

Press and hold the BOOT button
Connect the ESP32 over USB
Wait a few seconds
Release the BOOT button

Other development boards may not have a button, but all it really does is bridge GPIO0 and GND on the board which can be done with a wire, too. There are good docs on the connection process on the ESPHome website.

ESPHome Support #

ESPHome doesn’t natively support the IBS-P01B, but after some reading it seemed that the underlying messaging format/protocol is similar enough to the IBS-TH1 and IBS-TH2 that the same configuration can be used.

I also came across this blog post where the author had manually created an ESPHome configuration that appeared to work, and with the exact same hardware.

Nonetheless - to my mind, simpler is better most of the time, so I set about creating a simple configuration using the supported ESPHome platform.

ESPHome Dashboard #

There are a few different ways to get started with ESPHome. You can “install” using Docker, which seems to be the preferred method (passing through /dev/ttyUSBx to the container for USB connectivity), or manually using Python packages.

I tried to follow the instructions to get started with Home Assistant, but it seems this is not compatible with “unsupervised” Home Assistant servers like mine.

The esphome package is readily available in nixpkgs, so I was able to get started and fire up the dashboard like so:

❯ nix run unstable#esphome -- dashboard .
2024-11-12 14:47:19,072 INFO Starting dashboard web server on http://0.0.0.0:6052 and configuration dir ....

Browsing to http://localhost:6052 yielded me the following page:

Creating a Firmware #

I proceeded to create a new device, and entered some basic information about my Wifi network:

This is where things got interesting (for me, at least!). It turns out that the ESPHome dashboard makes use of the Web Serial API to program the ESP32 chip over USB from the browser. Given that my board was already connected and in programming mode, this was pretty simple!

The first time I tried this, I got a permission error. After looking through the messages in my kernel’s ring buffer with journalctl -k, and looking at the permissions on the /dev/ttyUSB0 device, it seemed likely that the issue was the root:root ownership with limited permissions. I solved this by running the following:

1
2


sudo chown root:users /dev/ttyUSB0
sudo chmod g+rwx /dev/ttyUSB0

This changed the group of the serial device to the same group as my user, then gave the group read, write and execute permissions. I retried in the browser and the device was flashed with the new firmware, connecting it to my network! Once the device was online, it appeared as such in the ESPHome dashboard, and I cooked up the following configuration by studing the docs on the inkbird_ibsth1_mini sensor component:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44


substitutions:
 name: esphome-web-caf3b8
 friendly_name: Tub Monitor

esphome:
 name: ${name}
 friendly_name: ${friendly_name}
 min_version: 2024.6.0
 name_add_mac_suffix: false
 project:
 name: esphome.web
 version: dev

esp32:
 board: esp32dev
 framework:
 type: arduino

web_server:
improv_serial:
logger:
api:
ota:
 - platform: esphome

wifi:
 # Pulled in from a separate secrets.yaml
 ssid: !secret wifi_ssid
 password: !secret wifi_password

esp32_ble_tracker:
 scan_parameters:
 active: true
 continuous: true
 duration: 1min

sensor:
 - platform: inkbird_ibsth1_mini
 # Bluetooth MAC address of my IBS-P01B
 mac_address: <redacted>
 temperature:
 name: "Tub Temperature"
 battery_level:
 name: "Tub Monitor Battery"

A few points on the above:

The device setup wizard created a secrets.yaml which can hold secrets to be referenced with the !secret <secret name> syntax. This is the same format as used by Home Assistant.
Specifying web_server: enables a small embedded web server that can be used to check status (see below!).
I had to specify the MAC address of the pool thermometer, which I collected using the Bluetooth Inspector app for iOS.

I clicked Install in the top right of the window, and the firmware was compiled and flashed to the device! This time I was able to flash over Wifi since the device was now connected to my network as a result of the onboarding process:

After selecting “Wirelessly”, and waiting for the firmware to be flashed, the logs shortly started flowing in:

To double check things were working, I then browsed to the IP address of the ESPHome device, and was greeted with a simple page that shows the current state of the device:

Adding to Home Assistant was as simple as adding a new integration of type ESPHome and specifying the IP address of the device!

3D Printing an Enclosure #

The final part of my build was to create a small enclosure for the device that would house the antenna correctly. I found a nice model on Printables, which printed pretty fast on my BambuLab X1 Carbon printer.

The photo below shows the device in place on the window sill, with the hot tub just visible in the background:

Home Assistant Dashboard #

With all this in place, all that remained was tying the information into my Home Assistant dashboard:

The above shows the hot tub temperature, as well as the battery level of the IBS-P01 sensor. Beneath, I’ve added basic controls and measurements available through the Tapo Home Assistant integration, which allows me to quickly toggle the pump and heater, and see their daily energy usage at a glance.

If you look closely, you’ll see the room temperatures resulting from the Roth underfloor heating integration I wrote about previously.

Summary #

We’re super pleased with our tub - it was built by Andy from Rustic Tubs. If you’re considering one and you’re in the UK, you could do a lot worse. It’s beautifully crafted, and Andy was really helpful and communicative throughout the process.

This was my first foray into ESP32-based projects, and I was pleasantly surprised with how seamless the process was. The ESPHome docs were clear, and the no-hassle flashing through the browser was a nice way to get started.

I hope this post is useful to people who are just getting started, and feel free to reach out if I’ve missed something!

Writing a Home Assistant Core Integration: Part 2

Wed, 16 Oct 2024 00:00:00 +0000

Introduction #

In my last post I described the first steps toward creating a new Home Assistant integration for the underfloor heating system in my house. In that post I outlined in detail how I set about creating a Python client for the API provided by the underfloor heating controller vendor.

In this post, I’ll describe the development setup, project structure and contribution process for building and landing a Home Assistant Core integration. I don’t consider myself an expert here, but I’ve documented my journey here in the hope that my experience might be useful to potential future contributors.

The finished integration can be seen in the Home Assistant docs under the name touchline_sl, and the code can be found on Github.

Development Setup #

The Home Assistant documentation recommends the use of Visual Studio Code with a devcontainer. This was a very quick way to get a working environment up and running, especially given that I already use Visual Studio Code for most of my programming, so I was immediately familiar.

The repository provides some tasks to help get started, including a task named Run Home Assistant Core, which takes care of setting up the runtime environment, installing dependencies and starting the server. Neat!

There are also a set of pre-commit hooks set up to ensure you don’t make any common mistakes, accidentally violate the formatting/static-typing rules for the project, forget to update requirements.txt files, etc.

This turned out to be a really nice way to get started, and if you’re new to Home Assistant Core development, I’d recommend giving this “batteries-included” approach a go. If it’s not for you, the project provides manual setup instructions too.

Integration Basics #

According to the documentation:

[An] integration is responsible for a specific domain within Home Assistant. Integrations can listen for or trigger events, offer actions, and maintain states.

Where a domain is…

a short name consisting of characters and underscores. This domain has to be unique and cannot be changed.

Err, right… while this is an accurate statement, it’s perhaps not the most enlightening for the budding new integration developer! At their core, integrations are Python modules that take information about a given system (like an underfloor heating system, or a smart plug, or a light bulb) and represent information about them in a format compatible with one of Home Assistant’s archetypes for devices/sensors/platforms/entities.

In this case, we’ll be representing Climate Entities, which have the sort of properties you might expect - target_temperature, current_humidity, current_temperature, etc.

To use Home Assistant’s terms, in my setup:

The platform is the underfloor heating system
The platform consists of some devices, in this case the physical thermostats in each room of my house
The devices each represent one or more climate entities (humidity, temperature, etc.)

File Structure #

The basic file structure can be laid down with some scaffold tooling, but even in its finished state, my integration doesn’t have many files:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


.
├── __init__.py # the component file
├── climate.py # ties info from the api into home assistant terms
├── config_flow.py # defines the fields/flow for integration config
├── const.py # constants used across the integration
├── coordinator.py # data update coordinator
├── manifest.json # defines project dependencies and metadata
├── strings.json # defines strings displayed in various ui elements
└── translations # a directory containing one file per language
 └── en.json # english translation of strings.json

And to give an idea of the scale of the project in its completed form:

----------------------------------------------------------------------
Language files blank comment code
----------------------------------------------------------------------
Python 5 71 27 215
JSON 3 0 0 82
----------------------------------------------------------------------
SUM: 8 71 27 297
----------------------------------------------------------------------

`manifest.json` #

Starting with the most simple first! The manifest.json describes the integration: what its name is, where its documentation is found, who owns the code and the libraries it depends on:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


{
 "domain": "touchline_sl",
 "name": "Roth Touchline SL",
 "codeowners": ["@jnsgruk"],
 "config_flow": true,
 "documentation": "https://www.home-assistant.io/integrations/touchline_sl",
 "integration_type": "hub",
 "iot_class": "cloud_polling",
 "requirements": ["pytouchlinesl==0.1.8"]
}

Here I selected cloud_polling as the iot_class, because my integration reaches out periodically to the API, polling for new information. You’ll note also that the integration requires my pytouchlinesl library in order to run.

`init.py` #

Typically for integrations, this file defines how to setup the integration, and how to unload it. You can see the full source of my implementation on Github.

I define an async_setup_entry() method which takes care of:

Establishing a coordinator per TouchlineSL module
Performing the initial hydration of data
Registering each TouchlineSL module as a device in Home Assistant’s Device Registry, which is where Home Assistant “keeps track of devices”

There is a shortened, annotated version of the method below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


async def async_setup_entry(hass: HomeAssistant, entry: TouchlineSLConfigEntry) -> bool:
 """Set up Roth Touchline SL from a config entry."""
 account = TouchlineSL(...)

 coordinators: list[TouchlineSLModuleCoordinator] = [
 TouchlineSLModuleCoordinator(hass, module)
 for module in await account.modules()
 ]
 # ...
 device_registry = dr.async_get(hass)
 # Create a new Device for each coorodinator to represent each module
 for c in coordinators:
 module = c.data.module
 device_registry.async_get_or_create(
 config_entry_id=entry.entry_id,
 identifiers={(DOMAIN, module.id)},
 name=module.name,
 # ...
 )

 entry.runtime_data = coordinators
 await hass.config_entries.async_forward_entry_setups(entry, PLATFORMS)
 return True

Before returning async_setup_entry() invokes async_forward_entry_setups(). This ensures that async_setup_entry() in climate.py is called to ensure each of the climate entities is registered.

The PLATFORMS variable is a list of platform types that the integration supports, in this case a single-item list containing just Platform.CLIMATE, which is how async_forward_entry_setups knows to invoke the async_setup_entry() method in climate.py:

1
2
3
4


from homeassistant.const import Platform
# ...
PLATFORMS: list[Platform] = [Platform.CLIMATE]
# ...

To initialise the integration, the user must authenticate with the Roth API so that module details are fetched before constructing the coordinator. Thus, before this code is executed, the user must go through the config flow…

`config_flow.py` #

Despite the final implementation being quite simple, this is probably one of the areas I found most challenging to get right. There are some docs but they only scratch the surface, and implementations in other integrations seem to vary quite dramatically (mostly depending on when they were written).

I went through a few iterations of this config flow, mostly because I had originally implemented the ability to select a specific module from the user’s account. The review process guided me toward simply logging into the account, then enrolling each of the modules associated with the account - since users can always disable entities they don’t wish to manage in Home Assistant.

The config_flow.py defines which input fields need to be presented to the user, and then passes on the relevant information needed to set up the integration. In my implementation, the code:

Prompts the user for their username and password
Authenticates with the service and fetches the user’s unique ID
Registers that unique ID, aborting if the specified account has already been used
Creates a config entry in Home Assistant that stores the user’s credentials

The config flow also has some basic error handling that can distinguish the difference between poor credentials, networking issues, etc. The full implementation can be seen on Github, but the important parts are highlighted in the following snippet:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


# ...
class TouchlineSLConfigFlow(ConfigFlow, domain=DOMAIN):
 # ...
 async def async_step_user(
 self, user_input: dict[str, Any] | None = None
 ) -> ConfigFlowResult:
 # ...
 if user_input is not None:
 try:
 account = TouchlineSL(
 username=user_input[CONF_USERNAME],
 password=user_input[CONF_PASSWORD],
 )
 # Use the credentials to fetch unique user id
 await account.user_id()
 except RothAPIError as e:
 # ...
 else:
 # Set unique ID, abort setup if already used
 unique_account_id = await account.user_id()
 await self.async_set_unique_id(str(unique_account_id))
 self._abort_if_unique_id_configured()

 # Create a config entry containing the user's credentials
 return self.async_create_entry(
 title=user_input[CONF_USERNAME], data=user_input
 )

 return self.async_show_form(
 step_id="user", data_schema=STEP_USER_DATA_SCHEMA, errors=errors
 )

What caught me out was how the fields are given titles/descriptions. These attributes are all configured in the strings.json, where the config map contains keys for each of the config “steps”.

The code above defines a step named user, since the method name is async_step_user. The step’s name, description and input fields are defined in the strings.json:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


{
 "config": {
 "flow_title": "Touchline SL Setup Flow",
 "error": {
 "cannot_connect": "[%key:common::config_flow::error::cannot_connect%]"
 // ...
 },
 "step": {
 "user": {
 "title": "Login to Touchline SL",
 "description": "Your credentials for the Roth Touchline SL mobile app/web service",
 "data": {
 "username": "[%key:common::config_flow::data::username%]",
 "password": "[%key:common::config_flow::data::password%]"
 }
 }
 },
 "abort": {
 "already_configured": "[%key:common::config_flow::abort::already_configured_device%]"
 }
 }
 // ...
}

The values displayed to the user are pulled from the translation files at runtime depending on their language configuration. In my integration, the corresponding translations/en.json contains the following fields that map to those defined above:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


{
 "config": {
 "flow_title": "Touchline SL Setup Flow",
 "error": {
 "cannot_connect": "Failed to connect"
 // ...
 },
 "step": {
 "user": {
 "data": {
 "password": "Password",
 "username": "Username"
 },
 "description": "Your credentials for the Roth Touchline SL mobile app/web service",
 "title": "Login to Touchline SL"
 }
 },
 "abort": {
 "already_configured": "Device is already configured"
 }
 }
 // ...
}

In hindsight, this is covered in the docs, but it definitely didn’t click with me when I was going through it, so it seemed worth calling out! The net result of this setup is a configuration dialog that looks like so:

`coordinator.py` #

This was an addition I made during the review process (more on that later), and appears to be the preferred way to implement the fetching of data from upstream APIs. By implementing a DataUpdateCoordinator, Home Assistant can ensure that a single coordinated poll happens across all entities managed by an integration. If an integration manages many entities for which it needs to fetch/update details, the coordinator helps ensure that the API is called only as often as is needed.

The coordinator class is very simple: it defines a single method _async_update_data which returns the data for the device its coordinating. As the developer, you can specify the type of the data returned by the coordinator. I chose to represent this as a Python dataclass:

1
2
3
4
5


@dataclass
class TouchlineSLModuleData:
 module: Module
 zones: dict[int, Zone]
 schedules: dict[str, GlobalScheduleModel]

The coordinator is initialised with some basic information such as a name and an update interval:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# ...
class TouchlineSLModuleCoordinator(DataUpdateCoordinator[TouchlineSLModuleData]):
 def __init__(self, hass: HomeAssistant, module: Module) -> None:
 super().__init__(
 hass,
 logger=_LOGGER,
 name=f"Touchline SL ({module.name})",
 update_interval=timedelta(seconds=30),
 )

 self.module = module

The _async_update_data method then queries the API, and returns data in the newly defined format:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


async def _async_update_data(self) -> TouchlineSLModuleData:
 try:
 zones = await self.module.zones()
 schedules = await self.module.schedules()
 except RothAPIError as error:
 # ...
 # Return the data using our dataclass from above
 return TouchlineSLModuleData(
 module=self.module,
 zones={z.id: z for z in zones},
 schedules={s.name: s for s in schedules},
 )

You can see the full implementation on Github.

`climate.py` #

And finally, on to the business logic of tying fields from the upstream API into the relevant attributes in Home Assistant!

The first task handled by this file is registering each of the climate entities by iterating over each zone, in each coordinator’s module:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# ...
async def async_setup_entry(
 hass: HomeAssistant,
 entry: TouchlineSLConfigEntry,
 async_add_entities: AddEntitiesCallback,
) -> None:
 """Set up the Touchline devices."""
 coordinators = entry.runtime_data
 async_add_entities(
 TouchlineSLZone(coordinator=coordinator, zone_id=zone_id)
 for coordinator in coordinators
 for zone_id in coordinator.data.zones
 )
# ...

Home Assistant entities have well-defined APIs - the docs for climate entities show the supported attributes and their data types. I used a combination of the docs, and the source code to establish how to implement my ClimateEntity, which boiled down to the following interface:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


class TouchlineSLZone(CoordinatorEntity[TouchlineSLModuleCoordinator], ClimateEntity):
 # Construct a Touchline SL climate zone.
 def __init__(self, coordinator: TouchlineSLModuleCoordinator, zone_id: int) -> None:
 # Handle updated data from the coordinator.
 @callback
 def _handle_coordinator_update(self) -> None:
 # Return the device object from the coordinator data.
 @property
 def zone(self) -> Zone
 # Report if the device is available.
 @property
 def available(self) -> bool
 #Set new target temperature.
 async def async_set_temperature(self, **kwargs: Any) -> None
 # Assign the zone to a particular global schedule.
 async def async_set_preset_mode(self, preset_mode: str) -> None
 # Populate attributes with data from the coordinator.
 def set_attr(self) -> None:

Arguably the most important part here is set_attr(). which takes care of mapping fields from the objects provided by my pytouchlinesl library to attributes on the climate entity:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


def set_attr(self) -> None:
 """Populate attributes with data from the coordinator."""
 schedule_names = self.coordinator.data.schedules.keys()

 self._attr_current_temperature = self.zone.temperature
 self._attr_target_temperature = self.zone.target_temperature
 self._attr_current_humidity = int(self.zone.humidity)
 self._attr_preset_modes = [*schedule_names, CONSTANT_TEMPERATURE]

 if self.zone.mode == "constantTemp":
 self._attr_preset_mode = CONSTANT_TEMPERATURE
 elif self.zone.mode == "globalSchedule":
 schedule = self.zone.schedule
 self._attr_preset_mode = schedule.name

The full implementation is on Github.

Testing #

Testing is (understandably) a little complicated. The requirements for landing code for an integration stipulate that you must include unit tests for any config/options flows. In my case, this meant the following:

Implementing a mock TouchlineSL client fixture to ensure that the unit tests don’t try to reach out to the real Roth API
Implementing a mock config entry fixture
A unit test for a successful config flow execution
A parametrised unit test for unsuccessful config flow due to possible exceptions when hitting the API
A unit test to ensure that multiple config flows resulting in the same user ID fail

To me this feels like the bare minimum, and unfortunately doesn’t really provide any confidence that the integration actually functions correctly. I’m hoping to improve this in the future, but for now further testing has been manual.

Docs & Brand #

As I was creating the Pull Request to contribute my integration, I was prompted by the template to link to further PRs that added the documentation and brand assets for my integration. I used the existing touchline docs as a template and modified them for my integration. The docs pull request added a single file touchline_sl.markdown containing 29 lines, which results in some nicely rendered docs on the Home Assistant website:

There were already branding assets for Roth as part of the touchlinesl integration that was previously merged. Based on some review feedback, I created a new Integration Brand named roth, with which both the touchline and touchline_sl integrations are associated. This has the nice effect of grouping them when setting up a new integration:

Contribution Process #

I submitted my first efforts for review in home-assistant/core#124557. The checklist guided me nicely through what needed to be done, and overall the process went pretty smoothly, and pretty quickly (despite the 117 comments!). The process took a little under two days in total. I was also fortunate with my timing, since my code landed the day before the next beta release was cut, so it shipped relatively quickly.

I’d like to offer my sincere thanks to @joostlek who not only reviewed my code extremely quickly, but took the time to explain things to me both in the Github PR, but also by proactively reaching out to me on the Home Assistant Discord, which I really appreciated.

I got a lot of feedback, which I expected. This was my first attempt at writing code for Home Assistant, and it’s a pretty large and well established project. I do think the project could do with some better developer documentation, which would dramatically reduce the burden of effort on reviewers and give contributors a better chance of “getting it right”.

One might argue I could have done more reading and more research - though I spent a good amount of time reading both the documentation and the source code of other integrations. I’m not sure I’d ever have reached the conclusions that @joostlek kindly nudged me toward.

Overall, my implementation was more brief, more simple and more efficient as a result of the review process, and based on my experience I’d advocate for having a go if you’ve been on the fence! Often submitting code to such a project can be daunting, but as with my experience when contributing to nixpkgs, if you go in with an open mind you’re sure to learn something from the process.

Results #

I’m really pleased with the results. However obtuse the developer experience felt at times, there is no denying that what I was able to get from my 297 lines of implementation code is quite staggering. I was super impressed that just by following the conventions I was able to get such intuitive controls, a full graphable history of temperatures and such a wide variety of ways to display the information in my various dashboards.

Once set up, you’re able to get a view of all the different zones imported by the module (the names of each zone are pulled from the upstream API if the zones are named, and each can be associated with a given area in Home Assistant):

The default dashboard displays thermostat controls for each zone. These allow you to see the current and target temprature, as well as adjust the target temperature if you need to:

Expanding the thermostat cards gives you a more detailed view, showing the mode and the “Preset”. My implementation maps “Presets” to “Global Schedules” configured in the Roth module:

Summary #

And that’s a wrap! I learned a bunch writing this integration, and the resulting user experience is quite a lot better than I get with the default Roth application.

My next mission is to use the information from these climate entities to automate opening the Velux windows in the roof when things get warm in the summer, but I’ve got a lot to learn about Home Assistant in the mean time!

Writing a Home Assistant Core Integration: Part 1

Wed, 11 Sep 2024 00:00:00 +0000

Introduction #

Back in March, my family and I moved into a new home. It’s a modern construction which came with solar panels (and associated inverter/battery storage), and uses an air source heat pump to heat the house with underfloor heating. Being a new renovation, nearly all of the appliances and components in the house have a form of internet connectivity (some more useful than others!).

Since day 1, I’ve been hoping to consolidate all of the various applications, data feeds and functions into one single place. I’ve been a long-time listener to the Self Hosted podcast, which often extols the virtues of Home Assistant. I’ve got no prior experience with Home Assistant, but for the last three months I’ve been running it on my home server, with a collection of custom integrations and hacks that enable me to control the underfloor heating and solar inverter.

The underfloor heating controller is a Roth Touchline SL system. In my set up, there is a single “module” which represents my house, and a number of “zones” which represent different rooms.

There was unfortunately no integration for this system in Home Assistant - there is one for the previous generation “Roth Touchline”, but this appears to function over the LAN, whereas the Touchline SL system is controlled over the internet using their API.

After reading the source code of a few other climate integrations in Home Assistant, it became clear to me that the first step was to create a Python client for the API which could be used in the integration.

This post will cover the design, implementation and limitations of the library I wrote: pytouchlinesl. If you came here to read about writing code for Home Assistant, you’ll have to wait for the next post! 😉

Designing the library #

Upstream API #

Usually, one would interact with a Roth Touchline SL system through their mobile apps, or through their online portal. The mobile app seems to be quite a lightweight wrapper around the web application, and I’ve not been able to detect any difference in functionality.

A bit of searching uncovered that Roth also provide an API for the Touchline SL system, and an OpenAPI spec. This made the process significantly easier, though there are some discrepancies in what is documented compared with how the API actually behaves. It feels to me like the API may have evolved, and the documentation has remained static - or perhaps it was always inaccurate? Either way, I spent quite a bit of time manually fuzzing the API to work out the correct set of parameters for some endpoints.

I also studied the web application using the Chrome Dev Tools. Of all the endpoints documented, it seemed like I’d only need the following:

POST /authentication: authenticates with the API, taking a username and password, and returning a token
GET /users/{user_id}/modules: returns a list of modules associated with the user
GET /users/{user_id}/modules/{module_udid}: returns details of a specific module (zones, schedules, etc.)

There is a slight awkwardness here, in that the last endpoint returns all of the data - for all zones, all schedules, etc. This feels inefficient, but I couldn’t find a way of getting information about a specific zone, or a specific schedule. The web application seems to rely upon polling the (undocumented) endpoint GET /users/{user_id}/modules/{module_id}/update/data/parents/[]/alarm_ids/[]/last_update/{timestamp}, which delivers deltas in the data since a given timestamp. This is useful in the context of the app because it can request the full dataset once, then request only changes from that point onwards, keeping the app state up to date without requesting the whole dataset.

Making changes to the configuration of zones and their temperatures is also fragmented. In essence, one can:

Set a zone to a constant temperature: POST /users/{user_id}/modules/{module_udid}/zones
Place a zone on a global schedule: POST /users/{user_id}/modules/{module_udid}/zones/{zone_id}/global_schedule
Place a zone on a local schedule: POST /users/{user_id}/modules/{module_udid}/zones/{zone_id}/local_schedule

The first is self-explanatory, enabling a zone to be set to a constant temperature (19.0C, for example). Touchline SL modules also support “schedules” which contain time periods for the specified zones to reach certain temperatures. In the case of a “Global Schedule”, multiple zones can be assigned, while a “Local Schedule” is specific to a single zone. The awkwardness in the API here is that to “add” a zone to a global schedule, you must re-specify the entire schedule, and specify all of the zones that should be on the schedule…

Basic Requirements #

In order to fulfil the basic functionality of my (future) Home Assistant integration, I limited the requirements of the first version to:

Authenticate with the API using a username and password
List modules associated with the account
Get a specific module
Get a specific zone
Get a list of global schedules
Get a specific global schedule
Assign a constant temperature to a zone
Assign a zone to a specific global schedule

I don’t use local schedules in my system, so I’ve omitted them for now, though updating the library to support them would be trivial.

Outline design/experience #

With those requirements in mind, I came up with a rough sketch of how I’d like the library to behave:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


tsl = TouchlineSL(username="foo", password="bar")
module = await tsl.module(id="deadbeef")

lounge = module.zone_by_name("Lounge")
kitchen = module.zone(1234)

# Properties should be available such as:
# lounge.current_temperature
# kitchen.humidity

await kitchen.set_temperature(temperature=20.0)

living_spaces = await tsl.schedule_by_name(schedule_name="Living Spaces")
await lounge.set_schedule(living_spaces.id)

And from that came a reasonable outline of the API for the library:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


class TouchlineSL:
 # Construct a class that represents a Touchline SL account
 def __init__( self, *, username: str | None = None, password: str | None = None)
 # Get a list of modules associated with the account
 async def modules(self) -> list[Module]
 # Get a specific module, by ID
 async def module(self, *, module_id: str) -> Module | None

class Module:
 # Get a list of zones from the module, optionally including disabled zones
 async def zones(self, *, include_off: bool = False) -> list[Zone]
 # Get a specific zone, by ID
 async def zone(self, zone_id: int) -> Zone | None
 # Get a specific zone, by name
 async def zone_by_name(self, zone_name: str) -> Zone | None
 # Get a list of global schedules
 async def schedules(self) -> list[Schedule]:
 # Get a specific schedule, by ID
 async def schedule(self, schedule_id: int) -> Schedule | None
 # Get a specific schedule, by name
 async def schedule_by_name(self, schedule_name: int) -> Schedule | None

class Zone:
 # Get the schedule the zone is assigned to
 def schedule(self) -> Schedule | None
 # Set the zone to a constant temperature
 async def set_temperature(self, temperature: float)
 # Assign the zone to a specific schedule
 async def set_schedule(self, schedule_id: int)

Note that after reading the code from other climate integrations in Home Assistant, it became clear to me that they favour the use of async libraries, and thus my library was designed to use asyncio from the start.

Python tools/libraries #

There are a couple of things I’ve found tiresome about Python over the years, but things do seem to be looking up. I’ve always found the package management and distribution to be awkward, and I can’t be the only one if the number of projects looking to target that problem is anything to go by (e.g. poetry, rye, uv, pdm, etc.).

Part of this seems to come from fractures in the community itself - there (still!) appears to be disagreements surrounding PEPs such as PEP-621 which introduced pyproject.toml as a way of managing project metadata and dependencies, with the maintainers of some high-profile and widely adopted libraries refusing to adopt it.

That said, there are a couple of things I’ve been meaning to try in anger, and this project was a good opportunity to do so:

`uv` #

Developed by Astral, uv is the “new shiny” at the time of writing, and I can understand why. Pitched as “Cargo, but for Python”, it aims to solve a myriad of problems in the Python ecosystem. uv can handle the download/install of multiple Python versions, the creation of virtual environments, running Python tools in a one-off fashion (like pipx), locking dependencies deeply in a project (by hash) and still maintains a pip compatible command-line experience with uv pip. To add to all of that, it’s ridiculously fast; on a couple of occasions I’ve actually found myself wondering if it did anything when installing dependencies for large projects, because it’s so much faster than I’m used to.

`pydantic` #

Pydantic is a data validation library for Python. It’s entirely driven by Python’s type-hints which means that you get nice integration with language servers. Pydantic allows you define data models in native Python, but emit standard JSON Schema docs for models. It’s integrated quite widely across the Python ecosystem, and to me feels like it bridges the gap between what I hoped type annotations would do for Python, and what they actually do in reality!

`ruff` #

I’ve been using this one for at least the last year for all my Python linting and formatting needs, but I still feel it deserves a call out. There have been a couple of small changes to command line API and such along the way, but overall I’ve found it to be a dramatic improvement over my last setup - which comprised of black, isort, flake8 and a pile of plugins. I had no particular beef with black, but I find flake8’s lack of pyproject.toml support irritating, and grew tired of plugins failing as flake8 released new versions.

In my experience, ruff is stupid fast, and because it ships flake8-compatible rules for all of the plugins I was using as one bundle, they never break. It’s also nice to just have one tool to use everywhere. If you’re interested, you can see how I configure ruff in the pyproject.toml.

Implementation #

With a basic design in mind, and tooling ready to go, I set about building the library itself. It was now time to reconcile my intended design with the realities of the provisions made by the upstream API.

I mentioned previously that the only useful endpoint for getting information about zones/schedules would in fact return all of the data for a given module. Too many calls to this endpoint would likely result in poor performance, so I wanted to introduce some basic caching along the way.

Client implementation #

For the underlying API client implementation, I opted for the following:

BaseClient: a class which inherits from Python’s abc.ABC. This enables the creation of multiple client implementations by defining of the set of methods/properties that any client interacting with the Roth API should define. This decision is primarily to support testing through dependency injection, rather than mocking with patches (more details on that later).
RothAPI: a concrete implementation of the BaseClient abstract class. It is here that I built the actual implementation of the client which handles authentication, GETing and POSTing data, caching, and marshalling API responses into the correct types (defined with Pydantic).

Also included in the client package is the models package. The models package contains (mostly) auto-generated Pydantic models based on real-life responses I got from the API. This is really a function of laziness, but was a convenient way to get type annotated models for the responses I was receiving from the API. Each time I hit a new endpoint, I took the JSON result and did a quick conversion with https://jsontopydantic.com/, before manually adjusting names and updating some fields with Literals.

Caching #

I mentioned earlier that I wanted to implement some basic caching. While I am aware of various plugins for aiohttp (and other request libraries) that could handle this for me, my requirements were quite simple, so I chose to just build it into the library. In this case, caching is implemented on the Module class. This is because the large blob of data that is requested to populate details about a module and its zones/schedules is requested per module.

The caching works like this:

The Module class has “private” attributes named _raw_data and _last_fetched:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


class Module:
 def __init__(
 self,
 *,
 client: BaseClient,
 module_data: AccountModuleModel,
 cache_validity: int = 30,
 ):
 # ...

 # Raw data about the zones, schedules, tiles in the module
 self._raw_data: ModuleModel
 # Unix timestamp representing the last time the _raw_data was fetched
 self._last_fetched = 0
 self._cache_validity = cache_validity

 # ...

There is only one method on this class that calls the underlying client, and that’s another “private” method named _data. This method takes an optional refresh keyword argument, which forces the _raw_data attribute to be updated, but by default will only fetch data if the cached data has expired (after the number of seconds specified in self._cache_validity). If refresh is false, and the cache hasn’t expired, it simply returns the stored raw data:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


async def _data(self, *, refresh: bool = False) -> ModuleModel:
 """Get the raw representation of the module from the upstream API.

 If the data has never been fetched from upstream, or the data is older
 than the cache validity period, then the data is refreshed using the
 upstream API.

 Args:
 refresh: (Optional): Force the data to be refreshed using the API.
 """
 if refresh or (round(time.time() * 1000) - self._last_fetched) > self._cache_validity:
 self._raw_data = await self._client.module(self.id)
 self._last_fetched = round(time.time() * 1000)

 return self._raw_data

Each of the public methods (zones(), zone(), schedule(), etc.) access the raw data through the _data() method, and pass through the refresh flag which is exposed. This means that any developer consuming this library can chose to live with the caching, or override it and force a refresh like so:

1
2
3
4
5
6
7


tsl = TouchlineSL(username="foo", password="bar")
module = await tsl.module(id="deadbeef")

# Request a zone, accepting cached data (default)
zone = await module.zone_by_name("kitchen")
# Or force the data to be refreshed using the upstream API
zone = await module.zone_by_name("kitchen", refresh=True)

Testing, CI & Publishing #

Testing #

In the previous section, I described how I’d set up an abstract base class for the API client, then created an implementation of that for the actual Roth API. One of the main reasons I like this pattern for API clients is that it simplifies testing and reduces the need for monkey patching or traditional mocking (yes, I know the fake API client is sort of a mock…).

Because TouchlineSL can optionally be constructed with any client that implements BaseClient’s abstract base class, it’s trivial to implement a fake API backend that returns fixture data to be used in testing. In this case, the fixtures are stored as JSON files in the repository, and contain real life responses received from the API.

The FakeRothAPI returns the sample data for each of the methods defined in the abstract class. The following is a partial extract from the fake client code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


# ...
from pytouchlinesl.client import BaseClient
from pytouchlinesl.client.models import AccountModuleModel, ModuleModel

data_dir = Path(os.path.realpath(__file__)).parent / "sample-data"


class FakeRothAPI(BaseClient):
 def __init__(self):
 self._user_id = 123456789
 self._token = "deadbeef"

 async def user_id(self) -> int:
 return self._user_id

 async def authenticated(self) -> bool:
 return True if self._token else False

 async def modules(self) -> list[AccountModuleModel]:
 with open(data_dir / "modules.json", "r") as f:
 data = json.load(f)

 return [AccountModuleModel(**m) for m in data]

#...

From there, I defined a number of test fixtures using pytest’s @pytest.fixture decorator, which provide an initialised TouchlineSL instance, backed by the fake client, a Module instance, and a Zone instance.

From there, the tests are fairly simple. Beyond injecting the fake client, there is no mocking required, which in my opinion keeps the test code much easier to read and understand. It also means I could focus more of my energy on validating the logic I was testing, rather than worrying about how patching might interact with the rest of the code. I’ve felt bad about mocking for years, and I think the clearest articulation I’ve seen is “Don’t Mock What You Don’t Own” in 5 Minutes.

One aspect of the test suite I don’t love is the use of time.sleep in certain tests. This is because my caching implementation relies on reading a timestamp to decide on whether it should refresh data. In general I steer away from sleeps in tests, as they’re often used to mask an underlying non-determinism, but in this case it felt a reasonable trade-off, given that I’m testing a time-based functionality.

CI #

I wanted to ensure that any pull requests were tested, and that they conform to the project’s formatting/linting rules. Since the project is hosted on Github, I used Github Actions for this. The pipeline for this project is pretty simple, it lints and formats the code with ruff failing if any files were changed by the formatter or any of the linting rules were violated.

Finally, uv is used to run pytest across a matrix of supported Python versions. I could have used uv to handle the download and install of different Python versions too, but the setup-python actions has served me perfectly well in the past.

Publishing #

Publishing is taken care of in CI. I don’t like having to remember the magic incantation for building, authenticating and publishing locally. I want the process to be as consistent and as transparent as possible for the people consuming the project.

What was new to me this time was publishing to PyPI with a “Trusted Publisher” setup. To quote their docs:

“Trusted publishing” is our term for using the OpenID Connect (OIDC) standard to exchange short-lived identity tokens between a trusted third-party service and PyPI. This method can be used in automated environments and eliminates the need to use manually generated API tokens to authenticate with PyPI when publishing.

Rather than setting up Github Actions to hold an API token in a Secret, there is some automation that links a Github project to a PyPI project. The project doesn’t even have to be previously published on PyPI to get started, and new projects can be configured as “Pending”, then published to for the first time from your CI system of choice 🚀.

I configured Github Actions to trigger the release of pytouchlinesl on new tags being pushed:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


# ...
publish:
 name: Build and Publish to PyPI
 runs-on: ubuntu-latest
 needs:
 - tests

 # Trusted publisher setup for PyPI
 environment:
 name: pypi
 url: https://pypi.org/p/pytouchlinesl
 permissions:
 id-token: write

 steps:
 - name: Checkout the code
 uses: actions/checkout@v4

 - name: Install `uv`
 run: |
 curl -LsSf https://astral.sh/uv/install.sh | sh

 - name: Build the package
 run: |
 uvx --from build pyproject-build --installer uv

 - name: Publish to PyPi
 uses: pypa/gh-action-pypi-publish@v1.10.0

Contributing to `nixpkgs` #

Finally, my Home Assistant server runs NixOS, so in order for the integration to be packaged easily in the future, I created a small PR to include pytouchlinesl in nixpkgs. The original PR went through pretty quickly - thanks to @drupol for the fast review!

Since then I’ve made a couple of minor version bumps to the library as I discovered small issues when building the integration, but the final derivation is quite compact (see below). The Python build tooling in Nix is quite mature at this point, and I gained a fair bit of experience using it when packaging snapcraft and charmcraft.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32


#...

buildPythonPackage rec {
 pname = "pytouchlinesl";
 version = "0.1.5";
 pyproject = true;

 disabled = pythonOlder "3.10";

 src = fetchFromGitHub {
 owner = "jnsgruk";
 repo = "pytouchlinesl";
 rev = "refs/tags/${version}";
 hash = "sha256-kdLMuxA1Ig85mH7s9rlmVjEsItXxRlDA1JTFasnJogg=";
 };

 build-system = [ setuptools ];

 dependencies = [
 aiohttp
 pydantic
 ];

 nativeCheckInputs = [
 pytestCheckHook
 pytest-asyncio
 ];

 pythonImportsCheck = [ "pytouchlinesl" ];

 # ...
}

Summary #

And that concludes the first part of this series! Hopefully you found this useful - I’ve learned a lot from people over the years by understanding how they approach problems, so I thought I’d post my methodology here in case it helps anyone refine their process.

I’m certainly no Python expert, so if you’ve spotted a mistake or you think I’m wrong, get in touch.

In the next post, I’ll cover writing the Home Assistant integration, and contributing it to Home Assistant.

Libations: Tailscale on the Rocks

Wed, 21 Aug 2024 00:00:00 +0000

Introduction #

I’m a long-time, self-professed connoisseur of cocktails. I’ve always enjoyed making (and drinking!) the classics, but I also like experimenting with new base spirits, techniques, bitters etc.

Over the years, I’ve collected recipes from a variety of sources. Some originated from books (such as Cocktail Codex and Cocktails Made Easy), others from websites (Difford’s Guide), and most importantly those that I’ve either guessed from things I’ve drunk elsewhere (like my favourite bar in Bristol, The Milk Thistle), or modifications to recipes from the referenced sources.

I wanted somewhere to store all these variations: somewhere that I could search easily from my iPhone (which is nearly always close to me when I’m making drinks). I wanted each recipe to fit in its entirety on my iPhone screen without the need to scroll.

Around the time I started thinking about this problem, I also learned about tsnet and was desperate for an excuse to try it out - and thus Libations was born as the product of two things I love: Tailscale and cocktails!

tswhat? #

Some time ago, Tailscale released a Go library named tsnet. To quote the website:

tsnet is a library that lets you embed Tailscale inside of a Go program

In this case, the embedded Tailscale works slightly different to how tailscaled works (by default, anyway…). Rather than using the universal TUN/TAP driver in the Linux kernel, tsnet instead uses a userspace TCP/IP networking stack, which enables the process embedding it to make direct connections to other devices on your tailnet as if it were “just another machine”. This makes it easy to embed, and drops the requirement for the process to be privileged enough to access /dev/tun.

One of the things I like about how tsnet presents applications as devices on the tailnet, is that you can employ ACLs to control who and what on your tailnet can access the application, rather than the device. I’ve solved this problem before by putting applications in their own systemd-nspawn container and joining those containers to my tailnet. Another nice option is tsnsrv which essentially acts as a Tailscale-aware proxy for individual applications, but in this case I wanted to bake it into the application - which I would only access over my tailnet.

Getting started with tsnet couldn’t be easier:

1
2
3


mkdir tsnet-app; cd tsnet-app
go mod init tsnet-app
go get tailscale.com/tsnet

That will get you set up with a basic Go project, with the tsnet library available. Create a new main.go file with the following contents:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


package main

import (
 "fmt"
 "log"
 "net/http"

 "tailscale.com/tsnet"
)

func main() {
 // Create a new tsnet server instance
 s := tsnet.Server{Hostname: "tsnet-test"}
 defer s.Close()

 // Have the tsnet server listen on :8080
 ln, err := s.Listen("tcp", ":8080")
 if err != nil {
 log.Fatal(err)
 }
 defer ln.Close()

 // Define a very simple handler with a simple Hello, World style message
 handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 fmt.Fprintf(w, "<html><body><h1>Hello from %s, tailnet!</h1>\n", s.Hostname)
 })

 // Start an HTTP server on the tsnet listener
 err = http.Serve(ln, handler)
 if err != nil {
 log.Fatal(err)
 }
}

This is about the most minimal example I could contrive. The code creates a simple instance of tsnet.Server with the hostname tsnet-app, listens on port 8080 and serves up a simple Hello, World! style message. On running the application you’ll see the following:

1
2
3
4
5


❯ go run .
2024/08/13 15:03:37 tsnet running state path /home/jon/.config/tsnet-tsnet-app/tailscaled.state
2024/08/13 15:03:37 tsnet starting with hostname "tsnet-test", varRoot "/home/jon/.config/tsnet-tsnet-app"
2024/08/13 15:03:38 LocalBackend state is NeedsLogin; running StartLoginInteractive...
2024/08/13 15:03:43 To start this tsnet server, restart with TS_AUTHKEY set, or go to: https://login.tailscale.com/a/deadbeeffeebdaed

Clicking the link will open a page in your browser that runs you through Tailscale’s authentication flow, after which you should be able to curl the page directly from any of your devices (assuming you’re not doing anything complicated with ACLs that might prevent it)!

1
2
3
4
5
6
7
8


❯ tailscale status
# ...
100.93.165.28 kara jnsgruk@ linux -
100.106.82.10 tsnet-test jnsgruk@ linux -
# ...

❯ curl http://tsnet-test:8080
<html><body><h1>Hello from tsnet-test, tailnet!</h1>

The library has a pretty small API surface, all of which is documented on pkg.go.dev.

Libations #

For my cocktail app, I wanted to employ a similar, albeit simplified, set of techniques that I use to build this blog. I love Go’s ability to embed static files that can be served as web assets.

Recipe Schema #

I wanted to represent recipes in a format that could be updated by hand if necessary, and easily parsed into a web frontend. I decided to use JSON for this, representing the recipes as a list:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


[
 {
 "id": 10,
 "name": "New York Sour",
 "base": ["Bourbon"],
 "glass": ["12oz Lowball"],
 "method": ["Dry Shake", "Shake"],
 "ice": ["Cubed"],
 "ingredients": [
 { "name": "Lemon Juice", "measure": "20", "unit": "ml" },
 { "name": "Sugar", "measure": "20", "unit": "ml" },
 { "name": "Red Wine", "measure": "10", "unit": "ml" },
 { "name": "Bourbon", "measure": "40", "unit": "ml" },
 { "name": "Egg White", "measure": "20", "unit": "ml" }
 ],
 "garnish": ["Lemon Sail"],
 "notes": "Use claret or malbec"
 }
]

This schema is able to capture all the relevant details from the different formats I’ve seen over the years. It would take some time to format my favourites into this schema, but that was always going to be the case.

I was fortunate enough to get access to the recipe collection from a well regarded cocktail bar in the UK. Unfortunately it was given to me in a hard-to-parse PDF, which resulted in many hours of playing with OCR tools and manual data cleaning - but enabled me to bootstrap the app with around 450 high quality recipes. I didn’t include their recipes in the libations repository, but I did include some of my own favourite concoctions in a sample recipe file. My Mezcal Margarita gets pretty good reviews 😉.

Server #

The server implementation needed to fulfil a few requirements:

Parse a specific recipes file, optionally passed via the command line
Have an embedded filesystem to contain static assets and templates
Be able to render HTML templates with the given recipes
Listen on either a tailnet (via tsnet), or locally (for testing convenience)
When listening on the tailnet, listen on HTTPS, redirecting HTTP traffic accordingly

I wanted to keep dependencies to a minimum to make things easier to maintain over time. The tsnet library pulls in a few indirect dependencies, but everything else Libations uses is in the Go standard library.

The recipes JSON schema is very simple, and is modelled with a couple of Go structs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


// Ingredient represents the name and quantity of a given ingredient in a recipe.
type Ingredient struct {
 Name string
 Measure string
 Unit string
}

// Drink represents all of the details for a given drink.
type Drink struct {
 ID int
 Name string
 Base []string
 Glass []string
 Method []string
 Ice []string
 Ingredients []Ingredient
 Garnish []string
 Notes string
}

The parseRecipes function checks whether or not the user passed the path to a specific recipe file, or whether it should default to parsing the sample recipes file. Once it’s determined the right file to parse, and validated its existence, it unmarshals the JSON using the Go standard library.

Users have the option of passing the -local flag when starting Libations, which bypasses tsnet completely and starts a local HTTP listener on the specific port. This makes for easier testing when iterating through changes to the web UI and other elements:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


// ...
addr := flag.String("addr", ":8080", "the address to listen on in the case of a local listener")
local := flag.Bool("local", false, "start on local addr; don't attach to a tailnet")

// ...
var listener *net.Listener
if *local {
 listener, err = localListener(*addr)
} else {
 listener, err = tailscaleListener(*hostname, *tsnetLogs)
}
if err != nil {
 slog.Error("failed to create listener", "error", err.Error())
 os.Exit(1)
}
//...

Setting up the tsnet server and listener is only mildly more complicated – but mostly due to my requirement that all HTTP traffic is redirected to HTTPS, using the LetsEncrypt certificates that Tailscale provides automatically. The redirects are handled by a separate Goroutine in this case:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


// Start a standard HTTP server in the background to redirect HTTP -> HTTPS.
go func() {
 httpLn, err := tsnetServer.Listen("tcp", ":80")
 if err != nil {
 slog.Error("unable to start HTTP listener, redirects from http->https will not work")
 return
 }

 slog.Info(fmt.Sprintf("started HTTP listener with tsnet at %s:80", hostname))

 err = http.Serve(httpLn, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 newURL := fmt.Sprintf("https://%s%s", r.Host, r.RequestURI)
 http.Redirect(w, r, newURL, http.StatusMovedPermanently)
 }))
 if err != nil {
 slog.Error("unable to start http server, redirects from http->https will not work")
 }
}()

With the correct listener selected, I create an http.ServeMux to handle routing to static assets and rendering templates, and pass that mux to the http.Serve method from the Go standard library - and that’s it! At the time of writing the Go code totals 235 lines - not bad!

Web Interface #

The web interface was designed primarily for mobile devices, and I’ve not yet done the work to make it excellent for larger-screened devices - though it’s certainly bearable. It’s also read-only at the moment: you can browse all the recipes, and there is a simple full-text search which can narrow the list of recipes down by searching for an ingredient, method, glass type, notes, etc.

As mentioned in an earlier post, I’m a big fan of the Vanilla Framework, which is a “simple, extensible CSS framework” from Canonical, and is used for all of Canonical’s apps and websites. Given that I had some prior experience using it, I decided to use it again here. I started using my tried and tested recipe of Vanilla + Hugo, but later reverted to using simple HTML templates with Go’s html/template package.

The result is a set of templates, which iterate over the recipe data from the JSON file, and output nicely styled HTML elements:

There is nothing fancy going on here - it’s nearly all stock Vanilla Framework. I do specify some overrides to make the colours a bit less Ubuntu-ish, but that’s it!

One detail I’m pleased with is the dynamic drink icons. The icons indicate the type of glass the particular drink should be served in. This is a simple trick: for each drink the HTML template renders a glass-icon partial, which reads the glass type specified in the recipe, and renders the appropriate SVG file which is then coloured with CSS.

Packaging for NixOS #

There were two main tasks in this category: creating the Nix package itself, and writing a simple NixOS module that would make it simple for me to run it on my NixOS server.

The project uses a Flake to provide the package, overlay and module. The standard library in Nix has good tooling for Go applications now, meaning the derivation is short:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


{
 buildGo122Module,
 lastModifiedDate,
 lib,
 ...
}:

let
 version = builtins.substring 0 8 lastModifiedDate;
in
buildGo122Module {
 pname = "libations";
 inherit version;
 src = lib.cleanSource ../.;
 vendorHash = "sha256-AWvaHyJL7Cm+zCY/vTuTAsgLbVy6WUNfmaGbyQOzMMQ=";
}

I haven’t cut any versioned releases of Libations at the time of writing - I’m using the last modified date of the flake to version the binaries.

The module starts the application with systemd, and optionally provides it with a recipes file. There are four options defined at the time of writing: services.libations.{enable,recipesFile,tailscaleKeyFile,package}:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


options = {
 services.libations = {
 enable = mkEnableOption "Enables the libations service";

 recipesFile = mkOption {
 type = nullOr path;
 default = null;
 example = "/var/lib/libations/recipes.json";
 description = ''
 A file containing drinks recipes per the Libations file format.
 See https://github.com/jnsgruk/libations.
 '';
 };

 tailscaleKeyFile = mkOption {
 type = nullOr path;
 default = null;
 example = "/run/agenix/libations-tsauthkey";
 description = ''
 A file containing a key for Libations to join a Tailscale network.
 See https://tailscale.com/kb/1085/auth-keys/.
 '';
 };

 package = mkPackageOption pkgs "libations" { };
 };
};

The tailscaleKeyFile option enables the service to automatically join a tailnet using an API key, rather than prompting the user to click a link and authorise manually.

These options are translated into a simple systemd unit:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


config = mkIf cfg.enable {
 systemd.services.libations = {
 description = "Libations cocktail recipe viewer";
 wantedBy = [ "multi-user.target" ];
 after = [ "network.target" ];
 environment = {
 "XDG_CONFIG_HOME" = "/var/lib/libations/";
 };
 serviceConfig = {
 DynamicUser = true;
 ExecStart = "${cfg.package}/bin/libations -recipes-file ${cfg.recipesFile}";
 Restart = "always";
 EnvironmentFile = cfg.tailscaleKeyFile;
 StateDirectory = "libations";
 StateDirectoryMode = "0750";
 };
 };
}

The XDG_CONFIG_HOME variable is set so that tsnet stores it’s state in /var/lib/libations, rather than trying to store it in the home directory of the dynamically created user for the systemd unit.

I use agenix on my NixOS machines to manage encrypted secrets, and for this project I used it to encrypt both the initial Tailscale auth key, and my super-secret recipe collection! The configuration to provide the secrets and configure the server to run libations is available on Github, but looks like so:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


{ config, self, ... }:
{
 age.secrets = {
 libations-auth-key = {
 file = "${self}/secrets/thor-libations-tskey.age";
 owner = "root";
 group = "root";
 mode = "400";
 };

 libations-recipes = {
 file = "${self}/secrets/thor-libations-recipes.age";
 owner = "root";
 group = "root";
 mode = "444";
 };
 };

 services.libations = {
 enable = true;
 recipesFile = config.age.secrets.libations-recipes.path;
 tailscaleKeyFile = config.age.secrets.libations-auth-key.path;
 };
}

As a result, the application is now available at https://libations, with a valid LetsEncrypt certificate, on all of my machines! 🎉

Summary #

This was a really fun project. It felt like a fun way to explore tsnet, and resulted in something that I’ve used a lot over the past year. I don’t have many plans to adjust things in the near future - though I do find myself wanting a nice interface to add new recipes from time to time.

And now for the twist: three weeks ago I gave up drinking alcohol (likely for good), so now I’m on a mission to find some non-alcoholic recipes to make life a little tastier! I suspect this will be hard work - and I’ll certainly miss some of my favourites, but my wife and I have already found some compelling alternatives.

If you’ve got a favourite recipe (alcoholic or not) and you liked the article, then perhaps open a PR and add it to the sample recipes file!

How I Computer in 2024

Wed, 31 Jul 2024 00:00:00 +0000

Since writing this post, I’ve posted an update about moving from NixOS to Ubuntu with more up to date content on my current setup.

Introduction #

I’m always fascinated to see how people use their computers - which applications they choose, how they set up their desktop environments and even how their screens are laid out on their desk. I’ve learned some great tricks from friends and colleagues over the years, so I thought I’d write up how I use my machines in 2024.

The setup I’m using today has been quite static for a couple of years, with only minor adjustments. Each time I change something significant, I leave it for at least a couple of months to try and build muscle memory and see if I’m going to make the adjustment permanent.

Hardware #

Desktop #

My main machine is a custom built desktop machine. It’s in a sombre looking, all black beQuiet Silent Base 600 case. I’ve never been into RGB lights - I’m much more into good thermals and silent operation. The full spec is as follows:

CPU: AMD Ryzen 9 7950X
GPU: AMD Radeon RX 7900XT
RAM: G.SKILL Trident Z5 Neo RGB 64GB DDR5-6000
PSU: Corsair HX1000i
Disk: 1TB SN850X + 2TB SN850X
Board: MSI MPG X670E CARBON WIFI
Cooler: beQuiet Dark Rock Pro 5
Case: beQuiet Silent Base 600 with 3x beQuiet Silent Wings 4 PWM fans
Keyboard: DURGOD Taurus K320 TKL with Cherry MX Brown switches
Mouse: Razer Deathadder V2 Pro
Monitor: 57" Samsung G95NC Odessey Neo G9
Camera: Sony ILME-FX3 / FE 28-70mm F3.5-5.6 / Elgato Cam Link 4K
Speakers: Audioengine A2+
Mic: RODE VideoMic GO II

On my desk, you’ll find a 57" Samsung G95NC Odessey Neo G9 monitor mounted on a gas spring arm, which is the newest addition to my setup. For 5 years, I’d been running a pair of 27" LG 27" UN850 4K monitors mounted on a dual monitor arm and had toyed with the idea of moving to an ultra-wide for a while. The Samsung display is the first I have found that doesn’t compromise on resolution - it’s the same resolution as my two LG monitors combined, but on a single panel. I must admit that I’m quite surprised how much of a productivity booster it is not having the split down the middle.

At the time of writing, I work for Canonical which is an all remote company. The combination of the company itself and my role as VP Engineering means I spend a good portion of my day on video calls. In my opinion, investing in a solid AV setup is a service to your colleagues, particularly where your role involves managing people. I’m currently running a Sony ILME-FX3 with the standard FE 28-70mm F3.5-5.6 lens, hooked up to an Elgato Cam Link 4K. For audio, I use a RODE VideoMic GO II and a pair of Audioengine A2+ speakers.

In the world of Linux desktops, I’ve found audio devices that present their own USB interface to be much less hassle. I completely disable the motherboard’s onboard sound, as well as the HDMI/DisplayPort sound outputs on my machine and leave just the USB audio interfaces from my mic and speakers enabled.

Server #

I use the term “server” loosely… my homelab has gone through many iterations over the years, from all “on-prem”, to a mix of cloud services and devices, and back again.

My current setup is very modest, partly because my workstation is such a monster, meaning I can easily spin up multiple VMs/containers there when I want to experiment and not really impact the performance of the machine for more routine tasks.

Most of my services run on a single Intel NUC6i7KYK. This machine has an Intel i7-6770HQ CPU, 16GB RAM and a 512GB Samsung 970 Pro NVMe drive internally. It’s connected to a Samsung 840 EVO 4TB SATA drive by USB. I’m not much of a data-hoarder so I don’t require too much storage.

This machine is getting a bit tired and I’m thinking about replacing it with something a little more modern later this year.

Laptop #

If I’m not at my desk, then I’m using my Lenovo Z13 Gen 1. I specified this machine with the AMD Ryzen 7 Pro 6860Z, 32GB RAM and a Hi-DPI display.

I can’t rate this machine highly enough. The build quality is a cut above even Lenovo’s normal standard - it feels very premium and much more in the style of Apple’s uni-body aluminium laptops. It’s got plenty of power, and the battery lasts most of the day under moderate usage.

I tend towards ultralight machines when I travel because I can always use my desktop machine remotely if I need more grunt (more on that later…), and I’m certainly not interested in trying to make dual integrated/discrete GPUs work properly.

Phone #

I carry an Apple iPhone 15 Pro. I’ve been an iPhone user since around 2011 and likely won’t change any time soon. My family all use iPhones (and therefore FaceTime) and I like the particular trade-off of convenience/privacy that’s provided by Apple - however flawed that might be in absolute terms. The phone works great with my Airpods, the camera is better than I am at taking photos, and the battery life seems pretty good too.

I wrap the phone in a Mous Limitless 5.0 Aramid Fibre case to avoid too many oops moments!

I find it difficult to get too excited about phones these days, I see them more as a commodity.

Connectivity & Security #

In 2021 I started using Tailscale in place of my hand-rolled Wireguard setup, and I haven’t looked back. It has to be one of my favourite pieces of technology ever. It runs on all of my things - desktops, laptops, servers, phones, tablets, etc.

I also recently took advantage of their partnership with Mullvad. I’ve used Mullvad as my default VPN provider when using untrusted networks for a few years - but using it through Tailscale means I can still access my tailnet while my internet traffic egresses through Mullvad without any extra configuration.

I use NextDNS as an alternative to running a Pi-Hole or similar. Tailscale have a nice integration which means that all the devices on my tailnet automatically get DNS-over-HTTPS without any additional configuration, as well as DNS-level ad-blocking.

I’ve got a couple of shared nodes in my tailnet - including one that my family can use as an exit node when they travel. As a result, I make quite extensive use of Tailscale ACLs to ensure people can only access what I want them to.

I’ve been a 1Password user for more than a decade now and I think their products are fantastic. Their Linux app sets the bar for modern cross-platform applications in my view. I recently started using their secrets capability at the CLI - the ability to store a .env file with a bunch of benign secret references, and have the actual secrets injected into the environment or a config file is very handy.

I also have a small collection of Yubikeys with different connectors. One lives on my desk attached to my desktop, another lives in my pocket or otherwise on my person, and another is in a safe. They’re all NFC enabled so they work nicely with my mobile devices. I configure my Yubikeys with ed25519 resident keys for SSH, along with storing my GPG key (which rarely gets used these days…).

One of my favourite things about the Yubikey is their ability to store TOTP codes. It’s a bit of a pain when I onboard a new account having to add the new secret to each key, but the upside is I don’t have to work out how to update/transfer them all each time I get a new phone! It’s also handy on the desktop to be able to run ykman oath accounts code <name>.

Productivity Apps #

A lot of my work is done in a browser. Canonical uses Google Workspace for emails, documents, slides, etc., so my default mode since joining has been to use Google Chrome for work things, and Firefox for personal things. I know that Firefox has Account Containers and other features that would help segregate the two, but I’ve found keeping my work and personal concerns in completely separate browsers to be useful.

After having run a Nextcloud server for several years on a Droplet (using docker-compose), I ultimately realised that I was only using the file syncing capability, and wasn’t moving much data even then. I switched to using Syncthing to avoid the overhead of running a server instance, which works particularly well when combined with Tailscale.

All of my notes, both work and personal, live in Obsidian. When I first discovered Obsidian I fell for the classic trick of installing all the extensions, and have since paired that back. I went very deep with Dataview, using it to collate actions from across my vault into various categories (meeting agendas, personal, reviews, etc.), but I found that as my vault grew the performance suffered quite a lot. A few months ago, I removed dataview, did some painful refactoring of my notes (lots of sederry and greppery!) and reverted to using Obsidian’s embedded search queries.

I tried to get into Zettelkasten but found the maintenance a little… boring? I’ve ended up with a simple structure that I find really helps me in my day-to-day at work. Each day gets its own “Daily Note” which includes my agenda, linking to ongoing notes with the people or regular meetings I’m in. The daily notes are also a place for me to collate loose notes which might be searched later:

The agenda and the links are automatically generated using a small Go application I wrote - this application scrapes my Google Calendar, and according to some rules and the knowledge it has of my vault, generates the Markdown for the agenda and copies it to the clipboard. Each day, I sit down and type agenda at the command line, then paste into Obsidian. The notes for each person contain a running log of my notes with that person or group by date.

I use a few Obsidian plugins to help here - including Templater, QuickAdd, Omnisearch and Linter. The first two are particularly handy for quickly inserting common meeting agendas, sets of interview questions, playbooks, etc.

I moved away from tracking tasks in Obsidian, and started using Todoist late last year. Todoist is great - I like to keep running lists of tasks per person, so that when I next meet them in a 1:1 or otherwise, I have a quick reference of all the things I’m meant to speak with them about - and I can achieve that very easily with Todoist labels. The Obsidian integration means I can integrate the agenda with the meeting note for a specific person:

Development #

I’ve been a long-time user of Alacritty as a terminal emulator. I mostly use Visual Studio Code on the desktop - I like the community support for plugins, themes, etc. I’m also pretty handy in vim - I still have quite a snazzy Neovim setup which I use whenever I’m at the terminal. You can see my neovim config on Github - I don’t go too wild on plugins, but I’ve come to like lightline, telescope, and nvim-tree-lua.

I mostly drive git from the command line, but I’ve recently taken to using Sublime Merge for complicated rebases, or where I want to stage lots of small hunks in files. I was a dedicated user of Sublime Text for some years, but felt like it lagged behind Visual Studio Code on features after a while - despite being somewhat addicted to how lightning fast Sublime Text felt in comparison.

OS / Desktop #

If you’ve read my blog before, it’ll be no surprise to you that I’m all-in on NixOS for all the things. I started that journey around 2 years ago and haven’t looked back. My journey on the Linux desktop has been quite varied over the years: my first ever Linux desktop experience was with Knoppix back in 2003. I then spent a few years dabbling with the various releases of Ubuntu before starting to use Linux on the desktop full-time in around 2014. From there I spent years on Arch Linux swapping between Plasma and GNOME about every 12 months.

I’ve become a fairly dedicated tiling window manager user, though I’ll admit that I bounced off it a few times before it stuck. When I made the switch to Sway in 2021, something clicked and I’ve not gone back from tiling since. I stuck with Sway in various configurations for quite a while, before moving to an almost identical looking setup based on Hyprland around 15 months ago. Hyprland seems nice - it’s mostly stable and I like the eye-candy.

Absolutely everything is themed with Catppuccin Macchiato. Not only do I love the theme, but I love how pervasive it is across all the apps/tools I use - and I’m a sucker for consistency!

You can see all the gory details of my Hyprland, waybar, rofi, mako, etc. on Github.

Server / Homelab #

My server machine also runs NixOS, with a collection of media services and utilities. Where possible all of the services are run as “native” NixOS modules, with some running inside systemd-nspawn containers using the built-in language for NixOS Containers. If I’m experimenting with a new service, I sometimes run them in Docker to start with, especially if there isn’t already a NixOS module and I want to decide whether or not to invest the time in writing one!

I run Caddy as a reverse proxy (recently switched from Traefik). It can talk directly to the Tailscale daemon to issue LetsEncrypt certs for devices on your tailnet. This Caddy instance acts as a reverse proxy onto all the services running on the server, along with some other services on my home LAN, all over TLS. I tend to access each of these services through Homepage (which I previously blogged about):

Each night, the contents of my iCloud Photos library is dumped using icloud-photos-downloader so that I have a local (and backed up) copy of my photos should anything untoward ever happen to my iCloud account.

It also runs a Home Assistant instance which runs my (in-progress!) custom integration for the underfloor heating and solar inverter in my house. I haven’t yet spent enough time with Home Assistant, but I plan to get it better set up over the coming months. I recently moved to a house with lots of “smart” devices, and I’d like to bring control of all the various devices into a single application. Once my experimenting is done, I’ll probably move the Home Assistant deployment to a dedicated low-power device.

This machine’s data is backed up nightly to Borgbase. As mentioned above, I use Syncthing to move files around, and I configure this server to act as a “receive only” target for all the directories that I sync. This means that my data is always in at least three places: on my desktop or laptop, on my server, and backed up to Borgbase. Sometimes I’ll ad-hoc access files that aren’t synced to a given machine using Files, which is a nice looking, single PHP-file gallery for your files. I keep meaning to replace this with something that isn’t PHP, but I’ve yet to find a more compelling blend of simplicity and compelling user experience.

Summary #

I don’t know how many other people are interested in how other people use their computers - but I hope you enjoyed the article. Feel free to reach out if you think I could be doing something better, or if you think you’ve got a killer app I might enjoy using!

Workstation VMs with LXD & Multipass

Tue, 25 Jun 2024 00:00:00 +0000

Introduction #

Over the years, I’ve used countless tools for creating virtual machines - often just for short periods of time when testing new software, trying out a new desktop environment, or creating a more isolated development environment. I’ve gone from just using the venerable qemu at the command line, to full-blown desktop applications like Virtualbox, to using virt-manager with libvirt.

When I joined Canonical back in March 2021, I’d hardly used LXD, and I hadn’t ever used Multipass. Since then, they’ve both become indispensable parts of my workflow, so I thought I’d share why I like them, and how I use each of them in my day to day work.

I work for Canonical, and am therefore invested in the success of their products, but at the time of writing I’m not responsible for either LXD or Multipass, and this post represents my honest opinions as a user of the products, and nothing more.

Installation / Distribution #

Both LXD and Multipass are available as snap packages, and that’s the most supported and recommended route for installation. LXD is available in the repos of a few other Linux distributions (including NixOS, Arch Linux), but the snap package also works great on Arch, Fedora, etc. I personally ran Multipass and LXD as snaps on Arch Linux for a couple of years without issue.

If you’d like to follow along with the commands in this post, you can get setup like so:

1
2
3
4
5
6
7
8
9


sudo snap install lxd
sudo lxd init --minimal

# If you'd like to use LXD/LXC commands without sudo
# run the following command and logout/login:
#
# sudo usermod -aG lxd $USER

sudo snap install multipass

Early on in my journey with NixOS, I packaged Multipass for Nix. I still maintain (and use!) the NixOS module. This was my first ever contribution to NixOS – a fairly colourful review process to say the least…

The result is that you can use something like the following in your configuration, and have multipass be available to you after a nixos-rebuild switch:

1
2
3


{
 virtualisation.multipass.enable = true;
}

LXD has been maintained in NixOS for many years now - and around this time last year I added support for the LXD UI. The screenshots you see throughout this post are all from LXD UI running on a NixOS machine using the following configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


{
 virtualisation.lxd = {
 enable = true;
 zfsSupport = true;
 ui.enable = true;
 };

 networking = {
 firewall = {
 trustedInterfaces = [ "lxdbr0" ];
 };
 };
}

Ubuntu on-demand with Multipass #

Multipass is designed to provide simple on-demand access to Ubuntu VMs from any workstation - whether that workstation is running Linux, macOS or Windows. It is designed to replicate, in a lightweight way, the experience of provisioning a simple Ubuntu VM on a cloud.

Multipass makes use of whichever the most appropriate hypervisor is on a given platform. On Linux, it can use QEMU, LXD or libvirt as backends, on Windows it can use Hyper-V or Virtualbox, and on macOS it can use QEMU or Virtualbox. Multipass refers to these backends as drivers.

Multipass’ scope is relatively limited, but in my opinion that’s what makes it so delightful to use. Once installed, the basic operation of Multipass couldn’t be simpler:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


❯ multipass shell
Launched: primary
Mounted '/home/jon' into 'primary:Home'
Welcome to Ubuntu 24.04 LTS (GNU/Linux 6.8.0-35-generic x86_64)

 * Documentation: https://help.ubuntu.com
 * Management: https://landscape.canonical.com
 * Support: https://ubuntu.com/pro

 System information as of Tue Jun 25 11:17:55 BST 2024

 System load: 0.4 Processes: 132
 Usage of /: 38.9% of 3.80GB Users logged in: 0
 Memory usage: 31% IPv4 address for ens3: 10.93.253.20
 Swap usage: 0%


Expanded Security Maintenance for Applications is not enabled.

3 updates can be applied immediately.
1 of these updates is a standard security update.
To see these additional updates run: apt list --upgradable

Enable ESM Apps to receive additional future security updates.
See https://ubuntu.com/esm or run: sudo pro status


ubuntu@primary:~$

This one command will take care of creating the primary instance if it doesn’t already exist, start the instance and drop you into a bash shell - normally in under a minute.

Multipass has a neat trick: it bundles a reverse SSHFS server that enables easy mounting of the host’s home directory into the VM. This happens by default for the primary instance. As a result the instance I created above has my home directory mounted at /home/ubuntu/Home - making it trivial to jump between editing code/files on my host and in the VM. I find this really useful - I can edit files on my workstation in my own editor, using my Yubikey to sign and push commits without having to worry about complicated provisioning or passthrough to the VM, and any files resulting from a build process on my workstation are instantly available in the VM for testing.

Multipass instances can be customised a little. You won’t find complicated features like PCI-passthrough, but basic parameters can be tweaked. The commands I usually run for setting up a development machine when I’m working on Juju/Charms are:

1
2
3
4
5
6


# Create a machine named 'dev' with 16 cores, 40GiB RAM and 100GiB disk
multipass launch noble -n dev -c 16 -m 40G -d 100G
# Mount my home directory into the VM
multipass mount /home/jon dev:/home/ubuntu/Home
# Get a shell in the VM
multipass shell dev

Once you’re done with an instance, you can remove it like so:

1
2


multipass remove dev
multipass purge

Multipass does have some more interesting features, though most of my usage is represented above. One feature that might be of more interest for MacOS or Windows users is aliases. This feature enables you to alias local commands to their counterparts in a Multipass VM, meaning for example that every time you run docker on your Mac, the command is actually executed inside the Multipass VM:

1
2
3


# Example of mapping the local `mdocker` command -> `docker` in
# the multipass VM
multipass alias dev:docker mdocker

Multipass will launch the latest Ubuntu LTS by default, but there are a number of other images available - including some “appliance” images for applications like Nextcloud, Mosquitto, etc.

There is also the concept of Blueprints which are essentially recipes for virtual machines with a given purpose. These are curated partly by the Multipass team, and partly by the community. A blueprint enables the author to specify cores, memory, disk, cloud-init data, aliases, health checks and more. The recipes themselves are maintained on Github, and you can see the list of available images/blueprints using multipass find:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


❯ multipass find
Image Aliases Version Description
core core16 20200818 Ubuntu Core 16
core18 20211124 Ubuntu Core 18
core20 20230119 Ubuntu Core 20
core22 20230717 Ubuntu Core 22
20.04 focal 20240612 Ubuntu 20.04 LTS
22.04 jammy 20240614 Ubuntu 22.04 LTS
23.10 mantic 20240619 Ubuntu 23.10
24.04 noble,lts 20240622 Ubuntu 24.04 LTS
daily:24.10 oracular,devel 20240622 Ubuntu 24.10
appliance:adguard-home 20200812 Ubuntu AdGuard Home Appliance
appliance:mosquitto 20200812 Ubuntu Mosquitto Appliance
appliance:nextcloud 20200812 Ubuntu Nextcloud Appliance
appliance:openhab 20200812 Ubuntu openHAB Home Appliance
appliance:plexmediaserver 20200812 Ubuntu Plex Media Server Appliance

Blueprint Aliases Version Description
anbox-cloud-appliance latest Anbox Cloud Appliance
charm-dev latest A development and testing environment for charmers
docker 0.4 A Docker environment with Portainer and related tools
jellyfin latest Jellyfin is a Free Software Media System that puts you in control of managing and streaming your media.
minikube latest minikube is local Kubernetes
ros-noetic 0.1 A development and testing environment for ROS Noetic.
ros2-humble 0.1 A development and testing environment for ROS 2 Humble.

The team also recently introduced the ability to snapshot virtual machines, though I must confess I’ve not tried it out in anger yet.

LXD… for VMs? #

For many people, LXD is a container manager - and indeed for many years it could “only” manage containers. LXD was built for running “system containers”, as opposed to “application containers” like Docker/Podman (or Kubernetes). Running a container with LXD is more similar to to running a container with systemd-nspawn, but with the added bonus that it can cluster across machines, authenticate against different identity backends, and manage more sophisticated storage.

Because LXD manages system containers, each container gets its own systemd, and behaves more like a ’lightweight VM’ sharing the host’s kernel. This turns out to be a very interesting property for people who want to get some of the benefits of containerisation (i.e. higher workload density, easier snapshotting, migration, etc.) with more legacy applications that might struggle to run effectively in application containers.

But this post is about virtual machines. Since the 4.0 LTS release, LXD has also supported running VMs with qemu. The API for launching a container is identical to launching a virtual machine. Better still, Canonical provides images for lots of different Linux distributions, and even desktop variants of some images - meaning you can quickly get up and running with a wide range of distributions, for example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# Launch a Ubuntu 24.04 LTS VM
lxc launch ubuntu:noble ubuntu --vm
# Get a shell inside the VM
lxc exec ubuntu bash

# Launch a Fedora 40 VM
lxc launch images:fedora/40 fedora --vm
# Get a shell inside the VM
lxc exec fedora bash

# Launch an Arch Linux VM (doesn't support secure boot yet)
lxc launch images:archlinux arch --vm -c security.secureboot=false
# Get a shell inside the VM
lxc exec arch bash

You can get a full list of virtual machine images like so:

1

lxc image ls images: --format=compact | grep VIRTUAL-MACHINE

LXD Desktop VMs #

Another neat trick for LXD is desktop virtual machines. These are launched with curated images that drop you into a minimal desktop environment that’s configured to automatically login. This has to be one of my favourite features of LXD!

1
2


# Launch a Ubuntu 24.04 LTS desktop VM and get a console
lxc launch images:ubuntu/24.04/desktop ubuntu --vm --console=vga

The guest is pre-configured to work correctly with SPICE, so that means clipboard integration, automatic resizing with the viewer window, USB redirection, etc. The same also works for other distros, as before:

1
2
3
4
5
6
7
8


# Launch an Arch desktop VM
lxc launch images:archlinux/desktop-gnome arch --vm \
 -c limits.cpu=8 \
 -c limits.memory=16GiB \
 -c security.secureboot=false

# Get a console using a separate command (if preferred!)
lxc console --type=vga arch

LXD UI 😍 #

Back in June 2023, Canonical announced early access to the LXD graphical user interface on their blog. The LXD UI is now generally available and enabled by default from LXD 5.21 onwards - though you can find instructions for enabling it on earlier versions in the docs. The summary is:

1
2
3
4


lxc config set core.https_address :8443

sudo snap set lxd ui.enable=true
sudo systemctl reload snap.lxd.daemon

In my opinion, the LXD UI is one of the best, if not the best way to interact with a hypervisor yet. Being a full-stack web application, it gains independence from different GUI toolkits on Linux and, provided the cluster is remote, can be accessed the same way from Windows, Mac and Linux.

I’ve used other hypervisors with web UIs, particularly Proxmox, and I’ve found the experience with LXD UI to be very smooth, even from the early days. The UI can walk you through the creation and management of VMs, containers, storage and networking. The UI can also give you a nice concise summary of each instance (below is the summary of the VM created using the command in the last section):

One of my favourite features is the web-based SPICE console for desktop VMs, which combined with the management features makes it trivial to stand up a desktop VM and start testing:

Why both? #

By now you’ve probably realised that LXD can do everything Multipass can do, and give much more flexibility - and that’s true. LXD is a full-featured hypervisor which supports much more sophisticated networking, PCI-passthrough, clustering, integration with enterprise identity providers, observability through Prometheus metrics and Loki log-forwarding, etc.

Multipass is small, lean and very easy to configure. If I just want a quick command-line only Ubuntu VM to play with, I still find multipass shell to be most convenient - especially with the automatic home directory mounting.

When I want to work with desktop VMs, interact with non-Ubuntu distributions, or work more closely with hardware, then I use LXD. I was already a bit of a closet LXD fan, having previously described it as a bit of a “secret weapon” for Canonical, but since the introduction of the LXD UI, I’m a fully paid up member of the LXD fan club 😉

Summary #

As I mentioned in the opening paragraphs - both LXD and Multipass have become central to a lot of my technical workflows. The reason I packaged Multipass for NixOS, was that I wanted to dive into daily-driving NixOS, but not without Multipass! In my opinion, the LXD UI is one of the most polished experiences for managing containers and VMs on Linux, and I’m really excited for what that team cooks up next.

Tracking Releases & CI Across Software Teams and Forges

Wed, 22 May 2024 00:00:00 +0000

Introduction #

In my day job at Canonical, I lead the teams developing Juju, and a whole host of charms. Charms are software packages used for deploying applications on any infrastructure you have available. The packages are portable, meaning you can use our PostgreSQL operator on AWS, on Azure, on Openstack, on Google Cloud, etc. We’re building up quite the portfolio of popular open source applications across data engineering, observability, identity, telco, MLOps and more. I won’t go into detail about Juju or charms in this post, but I likely will in a future post.

The important thing for this post is that I look after >10 software teams, who use two different software forges (Github and Launchpad), and all push artifacts into both the Snap Store and the Charmhub. I wanted a way to keep track of their releases, and provide a tool that my managers could use to do the same. Put simply, I wanted a unified view of:

The latest Github releases
The latest Launchpad tags
Channels in the Snap Store
Channels in Charmhub
CI status

In particular, I wanted an easy way to be able to tie releases/commits in a forge to specific revisions in our various stores. This would enable us to troubleshoot more easily when issues are reported in customer environments by making it easier to jump to the code for the specific revision they’re running.

At the time I started working on this, the effort at Canonical to build the portfolio of charms was really ramping up - and the only way of tracking which team owned which charm (between product teams, our IS department and our datacentre team) was a giant spreadsheet which was perpetually out of date (obviously!). The one thing that could be relied upon was which team owned a repo on Github or Launchpad - so my hope was to also reliably answer the question “which team owns the foo charm?”.

You can see the result of this effort at https://releases.juju.is, and a sneak peek below:

Prior Art #

I wasn’t the first to have these problems. The idea to start tracking information in this way came from the elementaryOS releases tracker. This was a tool they built for keeping track of their various repositories, and in particular those which had seen many commits since the last release. The code for the site is available on Github.

The elementaryOS tracker uses a Python script to scrape the Github API, which outputs a JSON representation of the state of their repositories. This is then parsed during the build of a Jekyll site, which is published on Github Pages using a Github Workflow.

This actually got me very close, and in fact my first attempt was a trivial fork of this project with a few slight modifications to the Python script. However, their tool wasn’t designed to be used across multiple distinct teams, and there are some limitations such as not rendering Markdown in the release notes.

I also wanted to make some stylistic changes. I’m not particularly familiar with Jekyll and have generally used Hugo for such tasks. I did maintain a Jekyll version for some months, but my lack of (recent) familiarity with the Ruby & gems ecosystem was making updates and maintenance more tedious than I liked.

Another tool I came across when I was looking to solve this was willow by Amolith. According to the README:

Willow helps developers, sysadmins, and homelabbers keep up with software releases across arbitrary forge platforms, including full-featured forges like GitHub, GitLab, or Forgejo as well as more minimal options like cgit or stagit.

This is super close - but it doesn’t have the snap/charm tracking I sought, and probably won’t do given the scope of the project.

`releasegen` #

As I started to modify the Python script used in the elementaryOS release tracker, it started to get quite large, and I became a little uncomfortable with the state of it. At the time I was reintroducing myself into Go after a couple of years out, and decided that I would essentially “start again” and write a tool for my release tracker from scratch.

What I came up with is releasegen. A terribly boring and unimaginative name that I originally intended to change, but never got around to! Nevertheless, releasegen solved a number of problems for me even in its first release. The first important change was support for separating releases across multiple Github organisations and teams, through the use of a simple config format:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


teams:
 - name: Frontend
 github:
 - org: acme-corp
 teams:
 - frontend-bots
 ignores:
 - some-old-project

 - name: Backend
 github:
 - org: acme-corp
 teams:
 - backend-engineers

 - name: Packaging
 launchpad:
 project-groups:
 - acme-corp-debs

You can see the config file used to generated releases.juju.is on Github.

The output format is heavily inspired by that of the original Python script, but has been enriched with a few more fields over time.

This tool is really quite simple: it takes a config file which points it at a combination of Github Orgs/Teams and Launchpad project groups, and outputs a big JSON file containing details of releases and associated packages. You can see an example of the output in the charm-eng-releases repo.

One of the main reasons I chose Go in the first place was because of it’s great support for concurrency. Data from Github is gathered using the google/go-github package, which provides an interface to the Github API. When I first started adding repos to the original Python version, the run time quickly grew to 10+ minutes. My intention was to spin up a goroutine per repository in releasegen, but I quickly ran into secondary rate-limits. It turns out that the Github API prohibits you from making too many concurrent requests.

For Launchpad, things are more complicated. There is an API but it lacks methods for grabbing tags, specific commits etc. I also wanted to avoid cloning all of the repos for which information is gathered. The Launchpad code browsing UI is based on cgit, and hasn’t fundamentally changed in a long time. For now, releasegen relies upon scraping the Launchpad web pages (using goquery) to get the information it needs. This is not ideal, but has been functioning better than you might expect for around 18 months. There’s also no limit on the requests that can be made to the web frontend of Launchpad - so processing is able to be done concurrently across repos in this case.

In more recent times, releasegen grew the ability to read badges out of project READMEs, and use those badges to link repos to a particular store (either the Snap store or the Charmhub). Support for reading badges and parsing Github CI badges was kindly contributed by one of my colleagues. I have subsequently generalised that initial implementation and added support for the Snap store too.

Parsing `releasegen` with Hugo #

Pleasingly, Hugo has the ability to read data from YAML, JSON, XML, or TOML files, which can then be rendered throughout the site.

Given that this tool was to be used for tracking releases from Canonical teams, I created a basic layout using the excellent Vanilla Framework. A layout in Hugo is nothing more than a collection of HTML/CSS/JS into which data can be rendered. The entry point for that in my project is this index.html. Here I layout the basic structure of the page, and use a number of partials to render common components across the site. There are very few adjustments to the standard Vanilla Framework style, and a couple of small Javascript files to provide tabs, modals, expanding table rows, table sorting and suchlike.

Aside from the big screenshot at the start of this post, I want to highlight a couple of details in the UI. Firstly, if there is an associated artifact for a given repository (a Snap or a Charm), the row can be expanded to show details of that artifact:

Each individual release can also be expanded using the “eye” icon, at which point a modal will be displayed containing the release notes, complete with links:

There is also some visual distinction between those repos on Github, and those on Launchpad:

Publishing with Github Actions #

The site is hosted on Github Pages, and if you’ve been following along, you’ll notice that the site would need to be “regenerated” for the information to remain fresh - Hugo is ultimately a static site generator. This problem is solved in my case with Github Actions. I created a workflow which is triggered every hour to dump the latest report using releasegen, regenerate the Hugo site, and commit the outcome to branch of the repo which is used to serve the page. The workflow itself is pretty simple, and can be seen on Github.

Summary #

This tool scratched an itch for me. It’s a little flawed in places: it doesn’t update in real time, releasegen could do with some ~~more~~ tests, and I’m not super proud of how it parses data about Launchpad projects. That said, it’s been fairly reliable over the past 18 months, and it does present information in a pretty consistent way (though I’m no designer!). I personally think it’s a good example of how things can be automated and simplified by combining existing tools in a short space of time.

I have an idea about how to generalise the processing of data, which would both remove my reliance on the Github API, and also unify the approach for gathering information about Git repos across forges (making it easier to support the likes of Codeberg and Sourcehut). I read an interesting blog recently about Serving a Website from a Git Repo Without Cloning It which implies that a lot of the information I require such as commits, tags, etc. could be gleaned directly from the git endpoint over HTTP, but I haven’t yet looked in detail.

It did solve the problem of keeping a big spreadsheet up to date as a means of tracking project ownership, and it’s certainly proved a useful tool for me to understand the relationship between commits, Github Releases and released revisions in our stores across teams. Some of my managers/seniors have found it really valuable, others not so much - this is generally reflected by how complete the information is for each team’s repos. Over the past few months, I’ve had a couple of teams reach out to me and ask to be added despite them not being part of my org, because they see it as a useful tool, so there’s that!

This tooling was also adopted by the Snapcrafters who run their own version of the dashboard. There are some subtle differences here - the UI in this case also displays each snap’s base (e.g. core18, core22, etc.). You can see the source code for that on Github.

That’s all for now! If you’ve built something similar or you think I’ve missed a trick, let me know!

Secure Boot & TPM-backed Full Disk Encryption on NixOS

Sat, 20 Apr 2024 00:00:00 +0000

Update: Since I wrote this post, I have been made aware of a particular situation where, at the time I write this (2025-01-17), the steps described in this article will result in a setup that is still (in many cases) vulnerable to an attack where the attacker has physical access to the machine. This may be acceptable in your threat model, but I’d encourage you to read the excellent article to gain a full understanding of the issue.

Introduction #

For the last decade (whoa…) or so, I’ve defaulted to using LUKS-encrypted drives for my machines. In general, I configure an unencrypted boot/EFI partition, then place either an ext4 or btrfs filesystem inside a LUKS container which is used for the root partition.

Some of my machines also have extra disks: my desktop has a 1TB NVMe drive for root, and a 2TB NVMe “data” drive mounted in my home directory under /home/jon/data. I don’t like having to type two different encryption passphrases at boot, so I usually have the extra disk automatically unlocked by putting the key in a file on the root drive, and placing an entry in /etc/crypttab.

This setup works fine for desktop machines, but it’s cumbersome on headless machines because unattended reboots require the disk passphrase to be entered at boot. Even then, all of my computers are exclusively used by me, and this setup means I have to enter two passwords on every boot to get to a working desktop environment (one for the disk, and one for the login manager).

I solved this recently with a combination of Secure Boot and a Trusted Platform Module (TPM), so let’s look at those first with a brief and high-level overview of each.

What’s a TPM? #

Most machines that have been manufactured in the last decade, and certainly in the last 5 years, contain a cryptographic coprocessor conforming to the Trusted Platform Module (TPM) spec. The TPM is a dedicated microcontroller primarily used for verifying the integrity of a machine.

TPMs can be used for storing cryptographic key material and performing basic cryptographic operations. The general premise is that keys can be loaded into the TPM, which enables the TPM to perform cryptographic operations using that key (signing, encrypting, etc.), but the key cannot be recovered or read from the TPM unless certain conditions are met. TPMs also provide other facilities such as secure random number generation, which in turn enables them to securely generate cryptographic keys.

Verifying system integrity essentially boils down to being able to ensure the machine hasn’t been tampered with between boots, and that the boot process itself hasn’t been compromised. A given firmware or operating system can take hardware “measurements” and store those measurements in dedicated slots called Platform Configuration Registers (PCRs). The measurements pertain to the underlying hardware and configuration of the machine. The TPM itself never performs the actual verification of the PCRs, and in fact has no knowledge of whether a measurement is inherently “good” or “bad”, but it can provide signed attestations of their values, which are then judged by the application requesting the attestation according to some policy.

Each PCR contains a hash representing a particular hardware measurement, which can be read at any time, but cannot be overwritten. Rather than allowing a traditional write operation, PCRs are updated through an “extend” operation which depends on the previous hash value, creating a chain of trust not dissimilar from how a blockchain is formed. This means that a given measurement can never be fully removed from the TPM.

Once the measurements are stored in the PCRs, there are various times and purposes for which the firmware or an operating system might read them - one example is for remote attestation during login to a system. In this scenario the attestation can be used to verify that the machine hasn’t been tampered with (perhaps in an Evil Maid attack). One could also store a Certificate Authority signing key in a TPM, and have the Certificate Authority software interface with the TPM to sign certificates using the PKCS#11 standard.

My use-case is to enabling the TPM to provide the passphrase to unlock a LUKS-encrypted disk, which is what I’ll focus on in this post.

Secure Boot #

Secure Boot is the mechanism by which the code executed by a machine’s Unified Extensible Firmware Interface (UEFI) can be verified as trusted. In the vast majority of cases, the first thing executed by the UEFI is a bootloader.

When Secure Boot is enabled, each binary executed by the UEFI must contain a checksum and a signature - which the UEFI verifies before launching the code. In the case that either the checksum or signature do not match, the UEFI will refuse the execute the code, and the boot process will halt.

Many OEM machines ship with Microsoft Windows installed, and thus ship with the necessary keys to validate signatures created with Microsoft’s certificate authority. Linux systems are able to utilise these keys through shim - a small and easily verifiable piece of software which is signed by Microsoft. shim sits between the UEFI and the bootloader in the boot process, obviating the need for every Linux bootloader to be signed by Microsoft on every release. The shim is designed to extend trust from the keys trusted by the computer’s firmware to a new set of keys controlled by the operating system.

But what does Secure Boot get us in reality? By signing the kernel, and in some cases a single UEFI PE binary known as a Unified Kernel Image (UKI) (which contains the bootloader, the kernel, the command-line used to boot the kernel, and other resources), one can be reasonably sure that the boot process hasn’t been tampered with.

This process thwarts a number of common physical attack vectors, such as manipulating the kernel command line to bypass the machine’s login and drop straight to a root shell - and combined with disk encryption can prevent offline data transfer from the machine. It also defends against malware which compromises the operating system’s boot process such that it can start before the OS and obfuscate it’s presence.

Threat Modelling #

As with any security measure, Secure Boot is not a silver bullet. You should always consider your own personal threat model, and the sorts of attacks you’re looking to defend against.

For example, Secure Boot can help prevent the sort of malware infections described in the previous section, but if an attacker gets physical access to your machine, and the UEFI isn’t adequately protected, they could simply disable secure boot and carry on unhindered.

I mitigate this by password protecting the UEFI. This isn’t perfect, but is likely sufficient protection for my threat model which is more about protecting chancers and petty thieves from gaining access to my information, than from determined attackers who gain physical access to my property.

Storing keys in a TPM is theoretically safe, in that each TPM has a unique seed which cannot be retrieved, and enables the TPM to deterministically generate keys between reboots. It’s very difficult to retrieve the seed, and thus very difficult to duplicate a TPM, but not impossible. Even then, the Linux kernel’s communication with the TPM on-the-wire is unencrypted, and the same can be said for many other subsystems which use the TPM. A recent example of this vulnerability was demonstrated by sniffing a Bitlocker key off the LPC bus in a Lenovo X1 Carbon laptop (using a Raspberry Pi Pico, no less). In many modern machines this is mitigated by the TPM being on-CPU, but the point still stands.

I choose to enroll Microsoft’s platform keys, which in theory degrades the security of my device in the case that Microsoft’s signing key is compromised, though all of my machines are compatible with fwupd and can receive updates to the database through that mechanism if required (and in fact have done in the past 18 months). This could be further mitigated by using custom keys and certificates for the full chain, but this is more overhead for daily operations and updates. It’s also worth considering whether you have the resources to fully secure your own chain - especially by comparison to Microsoft who spend tens of millions of dollars per year on security. If an attacker wants your information, and are able to compromise Microsoft’s CA, your own CA may not be such a hurdle.

Security measures are always a trade-off between Confidentiality, Availability and Integrity (CIA). In general, the more rigidly secure boot is implemented and configured, the more you’re protecting confidentiality and integrity. The choices I’ve made are slightly more in favour of availability, but nonetheless raise the bar for any attacker significantly.

Enabling Secure Boot on NixOS #

Now for the fun part! The process for enabling Secure Boot on NixOS has simplified in recent months owing to the creation of lanzaboote - a project which takes of preparing and signing Unified Kernel Images containing a custom stub, the bootloader, the Linux kernel, the kernel’s initrd and the kernel command line. lanzaboote also takes care of installing the UKI on the ESP partition so the UEFI can execute it at boot.

The lanzaboote stub differs slightly from systemd-stub, in that it doesn’t require the kernel and initrd to be part of the UKI. This is important for a generation-based operating system like NixOS because bundling the kernel and initrd into a new UKI for every generation would consume a lot of disk space, and quickly exhaust the ESP on most machines. In lanzaboote’s implementation, the kernel and initrd are stored separately on the ESP, and the chain of trust is preserved by validating the signature of the kernel, and embedding a cryptographic hash of the initrd into the signed UKI.

The project takes advantage of systems that have bootspec enabled, which is a relatively recent NixOS RFC that ensures configured machines maintain a file containing a set of memoised facts about a system’s closure. Bootspec aims to “provide more uniform feature support” to bootloaders in the NixOS ecosystem and “enable NixOS users to implement custom bootloader tools and policy” - of which lanzaboote is one.

Generating Secure Boot Keys #

The first step is to generate some keys for the secure boot process. This can be achieved using the sbctl package:

1
2
3
4


❯ sudo nix run nixpkgs#sbctl create-keys
Created Owner UUID 6ac34cc3-a23d-9745-ef33-a03f523d20a3
Creating secure boot keys...✓
Secure boot keys created!

This should only take a few seconds at maximum, and will result in a set of keys being populated in /etc/secureboot:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


❯ tree /etc/secureboot
/etc/secureboot
├── files.db
├── GUID
└── keys
 ├── db
 │ ├── db.key
 │ └── db.pem
 ├── dbx
 │ ├── dbx.key
 │ └── dbx.pem
 ├── KEK
 │ ├── KEK.key
 │ └── KEK.pem
 └── PK
 ├── PK.key
 └── PK.pem

Enable Bootspec #

Ensure that bootspec is enabled in your Nix configuration. You can see this in my flake for my desktop machine on Github:

1
2
3


{
 boot.bootspec.enabled = true;
}

Enable `lanzaboote` #

I use a flake to configure all of my machines, so I’m able to get access to lanzaboote by adding the upstream flake as an input to my own nixos-config flake:

1
2
3
4
5


# ...
inputs = {
 lanzaboote.url = "github:nix-community/lanzaboote";
};
# ...

Once you’ve added lanzaboote as a dependency, you’ll need to import the lanzaboote module:

1
2
3


# ...
imports = [ lanzaboote.nixosModules.lanzaboote ];
# ...

In my flake, I use a custom helper function to build NixOS configurations, so the module is passed directly to lib.nixosSystem through the modules attribute.

The lanzaboote module replaces the systemd-boot module, and as such you must explicitly disable systemd-boot when enabling lanzaboote. Additionally, if you wish to use the TPM for disk unlock (described in the next section), you must use the systemd initrd hooks (or something like clevis):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


boot = {
 initrd.systemd.enable = true;

 loader.systemd-boot.enable = lib.mkForce false;

 lanzaboote = {
 enable = true;
 pkiBundle = "/etc/secureboot";
 };
};

This is represented in my config here.

Once enabled, rebuild your system (in my case with sudo nixos-rebuild switch --flake /home/jon/nixos-config) and verify that your machine is ready for Secure Boot. Don’t panic about the kernel images being reported as not signed, this is expected:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


❯ sudo nix run unstable#sbctl verify
Verifying file database and EFI images in /boot...
✓ /boot/EFI/BOOT/BOOTX64.EFI is signed
✓ /boot/EFI/Linux/nixos-generation-414-376jna572gsb23snqs67t7s4bwxzb3epblmdnzweghuepopml2va.efi is signed
✓ /boot/EFI/Linux/nixos-generation-415-iqulgohymbdppgtxzho6ou3fcuxjbxhumpzm4vojmipwy3sbmuna.efi is signed
✓ /boot/EFI/Linux/nixos-generation-416-kxnzioafnduwwck3oypo7rqwtoat745czp2bpehoufp4yqiawypa.efi is signed
✗ /boot/EFI/nixos/kernel-6.8.2-242idodyvf36cpl6s5dskjy6mo4tjhszuwa3hye7qcjyuo5vnehq.efi is not signed
✗ /boot/EFI/nixos/kernel-6.8.5-zqulrwsucm6okcyns6v2jhh6fregk3bvsdth3yloqfymfbgnh64a.efi is not signed
✗ /boot/EFI/nixos/kernel-6.8.7-6mmixkr6ewywm5swgbi5ethbpgnyia4borzmkevcjx7n7t3mtida.efi is not signed
✓ /boot/EFI/systemd/systemd-bootx64.efi is signed

Prepare the UEFI #

Reboot your machine and enter the UEFI interface. This part of the process will vary from machine to machine depending on the UEFI implementation, but you’re looking to enable Secure Boot, and clear the preloaded Secure Boot keys. This may be referred to as “Setup Mode”, or erasing the “Platform Keys”.

While you’re here, I’d also advise setting a UEFI password before rebooting back into NixOS.

Enroll Secure Boot Keys #

The final stage in the process is to enroll your newly generated Secure Boot keys from step 1 into the UEFI. This is again achieved with sbctl:

1
2
3
4


❯ sudo nix run nixpkgs#sbctl enroll-keys -- --microsoft
Enrolling keys to EFI variables...
With vendor keys from microsoft...✓
Enrolled keys to the EFI variables!

I chose to use the --microsoft option to also enroll the UEFI vendor certificates from Microsoft. Some systems contain firmware that is signed and validated when Secure Boot is enabled, and omitting the Microsoft keys could prevent your device from booting - omit this option with caution!

Verify Secure Boot #

Once you’ve enrolled the keys, reboot the machine back into NixOS and use bootctl to confirm that Secure Boot is in fact enabled:

1
2
3
4
5
6
7


❯ bootctl status
System:
 Firmware: UEFI 2.80 (American Megatrends 5.26)
 Firmware Arch: x64
 Secure Boot: enabled (user)
 TPM2 Support: yes
 Boot into FW: supported

TPM Unlock of Root Partition #

Now that we’re (reasonably) confident that no one can tamper with the boot process, we can progress to allowing the machine to auto-unlock the encrypted disk using a key stored in the TPM.

This is actually more common than you might think - Windows has enabled this behaviour by default for some time with Bitlocker disk encryption, and Canonical is also working on bringing TPM-backed full disk encryption to Ubuntu.

This is probably the easiest step of them all! A simple invocation of systemd-cryptenroll is all that’s required. The arguments below instruct the machine that PCRs 0, 2, 7 and 12 should be measured and verified before the TPM is allowed to unlock the disk.

According to the Linux TPM PCR Regsitry, this means the following are measured before the LUKS key is presented:

PCR 0: Core system firmware executable code
PCR 2: Extended or pluggable executable code
PCR 7: SecureBoot state
PCR 12: Kernel command line, system credentials and system configuration images

1

❯ sudo systemd-cryptenroll --tpm2-device=auto --tpm2-pcrs=0+2+7+12 --wipe-slot=tpm2 /dev/nvme0n1p2

And that’s it! The next time you reboot, your disk should be automatically unlocked by the TPM, and your machine should boot straight to your display manager, or the TTY login if no display manager is configured.

Useful Resources #

None of the knowledge in this post is novel, but rather the culmination of some knowledge acquired over the past few years, and some more targeted reading more recently. In the process, I learned a bunch from the following:

Summary #

About 5 years ago, I was sporting a fully secure-boot enabled Dell XPS 13 running Arch Linux. Back then, the process was complicated, manual, and required a lot of maintenance between upgrades. For me, it was more pain than gain, but an interesting learning experience nonetheless.

When I sat down earlier this year to enable Secure Boot on NixOS, I’d set aside a few hours. I was astounded that 10 minutes later I was finished. I wrote this post as a memo to my future self, but also to illustrate how simple it can be to enable Secure Boot and TPM disk unlock in 2024.

I don’t claim to be an expert on the inner workings of TPMs, nor Secure Boot. The things I can say for certain are that TPMs are complex, that there are improvements that could be made to Linux’s interactions with the TPM, and that a determined and well-resourced attacker is likely going to succeed one way or another.

If you spot an inaccuracy in this post, reach out and let me know on Mastodon, on Telegram, by email, or however you prefer!

Until next time!

Update 2024/04/29: Thanks to @pimeys for pointing out that one must enable the systemd initrd hooks systemd-cryptenroll to function correctly, and also that PCR 12 must be measured to prevent the LUKS key from being released if the kernel command line has been modified.

Simplifying Test & Release of Snapped GUI Apps

Mon, 18 Mar 2024 00:00:00 +0000

Introduction #

For the past few months, I’ve been getting steadily more involved in Snapcrafters. The Snapcrafters are a community dedicated to the creation and maintenance of ~100 snap packages. They’re currently maintaining snaps for applications like Signal Desktop, Discord, Gimp, Terraform and many more, some with hundreds of thousands of weekly active users. As with any community organisation, maintainer participation can ebb and flow over time as people find themselves with competing priorities.

One of my personal goals for participation in the Snapcrafters org was to help them build more automated, sustainable processes for bumping versions of snaps as the upstreams move forward, and find a more robust way to test GUI applications before they’re released to the masses.

The snap store comes with a surprisingly rich delivery mechanism consisting of tracks, risks and branches. This means that (among other things) changes to applications can be tested by way of an incremental roll out by those willing to help out - by subscribing to the edge or candidate channels.

The Problem #

While bumping the versions of the applications in snap packages can be done trivially, testing that the new application can launch on the Linux desktop and function correctly is more difficult - especially given the “headless” nature of CI systems, and the inherent complexity of some of the applications involved.

Electron has, in my opinion, been a huge net win for the Linux desktop. Performance and early Wayland compatibility aside, the selection of mainstream applications available to the Linux desktop user is certainly much greater as a result. One downside is that each app is essentially a browser with lots of complex moving parts which can be difficult to maintain for packagers over time - and particularly so for a team of volunteers who may not be experts in the applications they help to maintain.

In a previous post I wrote about how KVM-acceleration is now available on Github Actions runners. While it’s certainly possible to just pull in various pieces of the Linux desktop using apt directly on a Github Actions runner, the resulting configuration normally involves convoluted setup with VNC or similar. Such setups are usually fragile, and can be more difficult to reproduce locally - making it slower to debug any issues that do arise.

LXD Desktop VMs #

Before working at Canonical, I must confess to having paid very little attention to LXD. Since joining, it’s become one of my most used tools in my daily workflow. I had some early experience with LXD a few years ago when it essentially “just” did Ubuntu containers, but it’s evolved into a very competent hypervisor in its own right, providing both container and virtual machine images for numerous different Linux distributions. In more recent history, desktop virtual machines were introduced which gives a very fast way to boot into a desktop across multiple distributions.

To boot into a Ubuntu 22.04 LTS desktop virtual machine, for example:

1

lxc launch images:ubuntu/22.04/desktop ubuntu --vm --console=vga

As a side note, last year Canonical announced the LXD UI, which is a beautiful web UI for managing clusters of LXD servers, and includes a graphical web console for virtual machines - which is incredibly useful for testing software across versions and desktops.

Wrapping LXD #

After playing with LXD a bunch locally, I confirmed that between lxc exec, lxc file pull I had all I need. The commands were all relatively simple, but I wanted to ensure that the Github Actions I wrote were as maintainable as possible, so I decided to write a small wrapper for LXD, which ended up being named ghvmctl (Github Virtual Machine Control) because naming is…. hard!

There is nothing special about ghvmctl, it is just a bash script. Perhaps one day I’ll implement it in something a little more… rigorous? That said, it’s wrapping relatively few shell commands, with very few variables, and it’s been solid for several months now. The sum of ghvmctl’s capabilities can be summarised as:

Launch Ubuntu desktop VMs
Dismiss any initial setup wizards
Ensure that gnome-screenshot is installed
Provide a simple way to install and run snaps
Provide a simple way to screenshot the whole screen, and the active window

The script is mostly contained in a single file, apart from ghvmctl-runner which is pushed automatically into any VMs started by ghvmctl, and provides a way for applications to be run with all the appropriate environment variables such that graphical applications can run when the VM is being controlled headlessly (such as DISPLAY, WAYLAND_DISPLAY, XDG_SESSION_TYPE, etc.).

There are very few dependencies for the script, but I decided to package it as a snap to simplify installing it on Github runners. The snap is simple, containing just the two scripts mentioned above, and the LXC client. This also means you can install and use the tool locally should you wish to experiment with it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


# Install ghvmctl
sudo snap install ghvmctl
# Allow ghvmctl to access the LXD socket
sudo snap connect ghvmctl:lxd lxd:lxd
# Launch a VM and prepare for testing
ghvmctl prepare
# Install a snap from the store
ghvmctl snap-install signal-desktop
# Run the snap on the desktop
ghvmctl snap-run signal-desktop

# Wait a few seconds for the app to start...

# Take screenshots and pull them back to $HOME/ghvmctl-screenshots
ghvmctl screenshot-full
ghvmctl screenshot-window

To simplify the installation and setup of ghvmctl, there is also a Github Action which takes care of enabling KVM on the runner, initialising LXD, installing ghvmctl and ensuring it has access to the LXD socket.

Building An Integrated Workflow #

By now, the Snapcrafters have quite a sophisticated collection of Github Actions which are used for managing the release lifecycle of snaps, but for me it was this piece that tied it all together for GUI applications. The actions can be summarised as:

Parsing snapcraft.yaml files for details such as name, architectures, version
Building snaps locally on Github Actions runners when Pull Requests are made
Building snaps across architectures on the Launchpad build farm when changes are merged
Create a “Call for Testing” Github Issue with details of the candidate revisions
Follow up on the issue with screenshots in a comment
Promote the snap to stable when a maintainer issues the command

None of these things are particularly hard in their own right, but they amount to some complicated juggling of Github Actions artefacts and tokens for various repositories and external services.

Building A Screenshot Action #

The Github Action responsible for collecting screenshots is used across multiple snaps to give maintainers a bit more confidence when releasing changes into the stable channels for their snaps.

The action makes use of ghvmctl to launch VMs, install the candidate snaps and collect screenshots of them. This turned out to be simple with the introduction of ghvmctl - the complicated part turned out to be where to store the screenshots such that they could be published in a comment! Github provides image/file hosting for comments on issue when the comments are made through the web UI. As far as I can tell, there is no way to submit a comment with an embedded picture from the Github API (let me know!), and I wasn’t keen to rely on a third party service such as Imgur.

The solution we settled on was to create a ci-screenshots, which could be published to by the workflows of each snap repository. We will, over time, clear out older screenshots.

End Result #

An example of the end result can be seen in this Github Issue. Let’s break down what happened.

First, once a change was merged into the Signal Desktop snap repository, and the resulting snap was built for each of its target architectures:

As part of this process an issue was automatically created containing information about the new candidate versions:

A couple of minutes later, the bot followed up with a comment containing screenshots of the application running on the desktop:

Once I was content that the snap was working correctly, and I’d tested the revision out locally, I then issued a command to promote the snap into the stable channel, where it was slowly rolled out across the user base:

You can see examples of this working across multiple snaps:

Summary #

We’re still in the process of refining and rolling out this process, but I hope that it will help reduce burden on maintainers over time, and result in fresher and more reliable desktop snaps in the Snap Store. If you’d like to get involved in Snapcrafters, reach out to me, post on the Snapcraft Discourse or join the Matrix room.

A homelab dashboard for NixOS

Tue, 05 Mar 2024 00:00:00 +0000

Introduction #

I run a very small homelab that provides some basic services to my home network. I’m not much of a data hoarder, but my lab consists of some redundant storage in a raidz2 ZFS pool, and I use the homelab as a receive-only target for Syncthing, and as the point from which backups of my critical data are made using Borg & Borgbase.

It also runs a few other small services - all of which are exclusively available over Tailscale to my other devices. I wanted a small dashboard solution that could give me links to each of those services with a nice simple URL.

There are certainly plenty of options; this seems to be a highly crowded space in the open source homelab world. I settled on the rather ambiguously named homepage. At the time of writing, my dashboard looks like so, though there are people who have been far more creative with the appearance!

Naturally, I wanted to run this on NixOS, so in July 2023 I landed one of my early contributions to the project in the form of PR #243094 which added the package (named homepage-dashboard), a basic NixOS module and a basic test.

Homepage’s Configuration #

Homepage is configured using a set of YAML files named services.yaml, bookmarks.yaml, widgets.yaml, etc. When I originally started writing the module, it would look for those config files in a hard-coded location (/config), and if the files were missing it would copy a skeleton config into place with some defaults to get you going.

The hard-coded location results from the fact that the upstream primarily support deploying Homepage using Docker. They expect the config directory to be bind-mounted into the container from the host. As part of the initial packaging effort, I contributed a patch upstream to allow customising this location by setting the HOMEPAGE_CONFIG_DIR environment variable, which I then set in the systemd unit configuration in the NixOS module to /var/lib/homepage-dashboard.

This has been working fine for a few months, but it’s been bugging me that my dashboard configuration is not part of the declarative system configuration. Once the initial skeleton had been copied in place, you were left to edit the files manually (and back them up) if you wanted to make changes. Moreover, homepage defaults to creating a logs directory as a subdirectory of the config directory. This makes some sense in a container environment, but given that homepage also logs to stdout (which is collected by the systemd journal on NixOS), it’s really just unnecessary duplication.

Evolving The Module Design #

Before writing the actual implementation of the module, I decided to first sketch out what I wanted my NixOS configuration to look like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


{
 services.homepage-dashboard = {
 # These options were already present in my configuration.
 enable = true;
 package = unstable.homepage-dashboard;

 # The following options were what I planned to add.

 # https://gethomepage.dev/latest/configs/settings/
 settings = {};

 # https://gethomepage.dev/latest/configs/bookmarks/
 bookmarks = [];

 # https://gethomepage.dev/latest/configs/services/
 services = [];

 # https://gethomepage.dev/latest/configs/service-widgets/
 widgets = [];

 # https://gethomepage.dev/latest/configs/kubernetes/
 kubernetes = { };

 # https://gethomepage.dev/latest/configs/docker/
 docker = { };

 # https://gethomepage.dev/latest/configs/custom-css-js/
 customJS = "";
 customCSS = "";
 };
}

Each of the new sections would then map neatly to the different configuration section in the upstream documentation. Based on my learning from the Scrutiny module, I wanted to utilise the same RFC42 approach which would obviate the need for the module to specify every possible supported configuration option, resulting in a large, difficult to maintain module which could quickly fall behind the upstream project.

Homepage supports a large number of widgets (see below) which are able to scrape information from the API of various devices and services. These often require an API key or token of some kind, and having those in plaintext as part of the machine configuration is undesirable from a security perspective - even if your services are all on a private network like mine. Luckily I found out (tucked away in the docs) that Homepage can inject secret values into the configuration using environment variables.

Given the template value of {{HOMEPAGE_VAR_FOOBAR}} as part of the configuration, Homepage will automatically substitute the value of the variable HOMEPAGE_VAR_FOOBAR.

I decided to provide a single configuration option named environmentFile so that users can supply the path to an environment file containing all of their variables. This file can be omitted from Git repositories and configurations, or included in encrypted form. I achieve this by including the file encrypted using agenix which integrates @FiloSottile’s wonderful age into NixOS. You can see how that’s supplied as part of my nixos-config.

Backwards Compatibility #

According to my relatively naive Github search I estimated that there are not that many users of the module - likely in the tens, rather than the hundreds or thousands. That said, I think its important not to break those users. There’s no reason to expect that a nix flake update should break your system.

The way I chose to handle this in the module was to check if any of the new config options are set. If they’re not, the module behaves as before, but displays a deprecation warning:

The implementation of this check is relatively crude, but it works, and it will only be around until the release of NixOS 24.05 (in May 24):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


config =
 let
 # If homepage-dashboard is enabled, but none of the configuration values have been updated,
 # then default to "unmanaged" configuration which is manually updated in
 # var/lib/homepage-dashboard. This is to maintain backwards compatibility, and should be
 # deprecated in a future release.
 managedConfig = !(
 cfg.bookmarks == [ ] &&
 cfg.customCSS == "" &&
 cfg.customJS == "" &&
 cfg.docker == { } &&
 cfg.kubernetes == { } &&
 cfg.services == [ ] &&
 cfg.settings == { } &&
 cfg.widgets == [ ]
 );

 configDir = if managedConfig then "/etc/homepage-dashboard" else "/var/lib/homepage-dashboard";

 msg = "using unmanaged configuration for homepage-dashboard is deprecated and will be removed"
 + " in 24.05. please see the NixOS documentation for `services.homepage-dashboard' and add"
 + " your bookmarks, services, widgets, and other configuration using the options provided.";
 in
 lib.mkIf cfg.enable {
 # Display the deprecation warning if the configuration isn't managed
 warnings = lib.optional (!managedConfig) msg;

# ...

Solving Log Duplication #

I mentioned in a previous section that Homepage logs to both stdout and a logs directory by default. While the log file path can be customised, it’s not currently possible to disable the file logging completely. It’s not desirable to have the log file in this context, because all of the logs are collected by systemd anyway.

Looking at the upstream implementation, the logger is instantiated and configured in a single logger.js file. Homepage has a policy that they won’t accept feature contributions (even if you do the implementation) unless the feature gets at least 10 upvotes. I filed a feature request, but it’s yet to get enough votes to be accepted.

In the mean time I wrote a short patch on a branch in my personal fork which makes Homepage adhere to an environment variable named LOG_TARGETS. The possible values are both, file or stdout with a default value of both to respect the existing behaviour and remain backward compatible. The patch is now applied in the Nix package as part of nixpkgs, and the module configures the systemd unit by setting the LOG_TARGETS variable to stdout in cases where the configuration is managed:

1
2
3
4
5
6
7
8
9


{
 # ...
 environment = {
 HOMEPAGE_CONFIG_DIR = configDir;
 PORT = toString cfg.listenPort;
 LOG_TARGETS = lib.mkIf managedConfig "stdout";
 };
 # ...
}

Update: My feature request got upvoted pretty quickly, and as a result my pull request was merged. This means that following the next release of Homepage, I’ll be able to drop the patch from the Nix package. Thanks to all those who upvoted it!

Bolstering The Test Suite #

When I originally implemented the tests for the module, they simply enabled the service and ensure that it responded on the specified port. I wanted to include some logic in the test that ensured the ability to detect when managed configuration should be used, and when the module should respect an existing implementation.

The NixOS test suite supports specifying multiple machines as part of a given test, so extending the previous implementation wasn’t particularly cumbersome. See below for the (annotated) implementation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49


import ./make-test-python.nix ({ lib, ... }: {
 name = "homepage-dashboard";
 meta.maintainers = with lib.maintainers; [ jnsgruk ];

 # Create a machine that uses the legacy module format,
 # where configuration is unmanaged by nix, and relies
 # upon YAML files.
 nodes.unmanaged_conf = { pkgs, ... }: {
 services.homepage-dashboard.enable = true;
 };

 # Create another machine that sets some simple
 # configuration using the new module system. This
 # doesn't need to be exhaustive, just enough to trigger
 # the condition that makes the module use managed config.
 nodes.managed_conf = { pkgs, ... }: {
 services.homepage-dashboard = {
 enable = true;
 settings.title = "custom";
 };
 };

 testScript = ''
 # Ensure the services are started on unmanaged machine,
 # and that the service responds to HTTP requests on the
 # expected port.
 unmanaged_conf.wait_for_unit("homepage-dashboard.service")
 unmanaged_conf.wait_for_open_port(8082)
 unmanaged_conf.succeed("curl --fail http://localhost:8082/")

 # Ensure that /etc/homepage-dashboard doesn't exist, and boilerplate
 # configs are copied into place in `/var/lib/homepage-dashboard`.
 # This validates the existing behaviour.
 unmanaged_conf.fail("test -d /etc/homepage-dashboard")
 unmanaged_conf.succeed("test -f /var/lib/private/homepage-dashboard/settings.yaml")

 # Ensure the services are started on managed machine,
 # and that the service responds to HTTP requests on the
 # expected port.
 managed_conf.wait_for_unit("homepage-dashboard.service")
 managed_conf.wait_for_open_port(8082)
 managed_conf.succeed("curl --fail http://localhost:8082/")

 # Ensure /etc/homepage-dashboard is created and unmanaged
 # conf location isn't present
 managed_conf.succeed("test -d /etc/homepage-dashboard")
 managed_conf.fail("test -f /var/lib/private/homepage-dashboard/settings.yaml")
 '';
})

This is by no means exhaustive, and I can certainly imagine increasing the coverage here at a later date, but it does at least give some confidence when working on the module that the two basic modes of operation are functioning correctly.

Migrating Existing Configurations #

If you have been using the module in its past form, you may be wondering what the easiest way to migrate to the new format is…

I made the shift using yaml2nix to convert my existing YAML configurations to Nix expressions, and then formatted the output using nixpkgs-fmt. For example, given the following settings.yaml (which came from my homelab before I moved over):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


---
# For configuration options and examples, please see:
# https://gethomepage.dev/en/configs/settings

title: sgrs dashboard
favicon: https://jnsgr.uk/favicon.ico
headerStyle: clean

layout:
 media:
 style: row
 columns: 3
 infra:
 style: row
 columns: 4
 machines:
 style: row
 columns: 4

You can do the following to get a Nix expression that can be assigned to services.homepage-dashboard.settings in your machine configuration, converting the YAML to a Nix expression:

 ~/temp
❯ nix run nixpkgs#yaml2nix settings.yaml
{ title = "sgrs dashboard"; favicon = "https://jnsgr.uk/favicon.ico"; headerStyle = "clean"; layout = { media = { style = "row"; columns = 3; }; infra = { style = "row"; columns = 4; }; machines = { style = "row"; columns = 4; }; }; }

With that output, you can insert a few line breaks and rely on nixpkgs-fmt to get everything lined up properly. You can see my complete dashboard configuration in Nix format as part of my nixos-config repository.

Summary #

The PR was merged earlier today, and will now need to trickle through the branches on its way to nixos-unstable. At the time of writing, it hasn’t quite made it there:

You can track for yourself on the nixpkgs tracker, but the time delay should give you a chance to migrate your configuration!

See Github for the full module implementation, package and tests.

Let me know if you’re using the module, or if you run into any issues! If you’re a fan of Homepage, then consider helping out with the project or sponsoring them on Github, and once again thank you to those who helped review and shape the module as part of this upgrade!

Contributing Scrutiny to nixpkgs

Mon, 19 Feb 2024 00:00:00 +0000

Introduction #

I’m quite overwhelmed by the number of people who read, and engaged with my last post on packaging Scrutiny for NixOS. I hadn’t intended on posting to this blog more than maybe once per month, but I decided to continue documenting the journey of landing Scrutiny upstream in nixpkgs.

One of the things that really struck me about NixOS early on was how simple the contribution process is. A single Github repository contains all the packages, all the modules, and all the tests.

I’ve found the community incredibly helpful and respectful in all of my contributions - often with 2-3 people quickly providing reviews to help shape the contribution. I’ve heard complaints that reviews can be quite pedantic, and while I agree to some extent, the reviews nearly always come with concrete suggestions which are a great way to learn. Notwithstanding the fact that you are contributing to an operating system package that could land on many thousands of machines - it’s important to be careful!

If you’ve been deliberating about whether or not to submit your package, module or otherwise, I’d encourage you to go for it. I’ve always found it rewarding and it’s taught me a lot about Nix and NixOS.

My first contribution was adding a package, module and test for multipass - as you can see I had plenty to learn, but people were generous with their time and it resulted in a contribution that I use daily to get my work done on NixOS.

For those losing sleep over the poorly disk I showed in my last post, you’ll be relieved to know that it’s now been replaced! Incidentally this was the first time I’ve replaced a disk in a ZFS array and I was super impressed with how easy the process was!

Improving My Work #

Over the weekend following my first post about Scrutiny, a couple of people identified some small issues and made pull requests to resolve them - thank you to those people! You can see their work here, here and here. Someone also pointed out to me that string-interpolating YAML wasn’t the most ideal solution to rendering the configuration file, and the I could have used builtins.toJSON, taking advantage of the fact that JSON is also valid YAML.

I also wanted to make sure that there was a way for people to use the module with an existing InfluxDB deployment. My original implementation assumed that it could just add services.influxdb2.enable = true to the system configuration and use the default credentials to connect. I added some options to the module for configuring a different InfluxDB instance if required. These were later modified slightly to use lib.types.nullOr lib.types.str following a helpful review comment.

Next up was refactoring the helper function I wrote for generating the configuration file. As mentioned above, Nix comes with a handy builtins.toJSON function which takes a native Nix expression (an attribute set in this case) and renders it to JSON. This obviates the need to string-interpolate YAML, which is error prone at the best of times. The renewed function looked like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


{
 mkScrutinyCfg = cfg: pkgs.writeTextFile {
 name = "scrutiny.yaml";
 text = builtins.toJSON {
 version = 1;
 web = {
 listen = {
 inherit (cfg) host port;
 basepath = if cfg.basepath != "" then "/" + cfg.basepath else "";
 };

 database = {
 location = "/var/lib/scrutiny/scrutiny.db";
 };

 src = {
 frontend.path = "${pkgs.scrutiny}/share/scrutiny";
 };

 influxdb = {
 inherit (cfg.influxdb) scheme host port token org bucket;
 tls.insecure_skip_verify = cfg.influxdb.tlsSkipVerify;
 };
 };
 log = {
 level = cfg.logLevel;
 };
 };
 };
}

I also noticed that I had hard-coded the path to the binaries in the rendered systemd units. You might recall that each Nix package contains meta block, which adds information about the package such as the license, maintainer, etc. For each of the packages, I added a mainProgram attribute which enables the use of something like lib.getExe pkgs.scrutiny rather than ${pkgs.scrutiny}/bin/scrutiny.

1
2
3
4
5
6
7


{
 # ...
 meta = {
 mainProgram = "scrutiny";
 # ...
 };
}

Finally, I wanted to simplify the packaging of the Go programs slightly. In the previous post I used buildPhase to override the default build behaviour of the buildGoModule function. I later discovered the subPackages attribute, which enabled me to achieve the same effect without overriding the process manually. You can see the commit with all the changes on Github, but below is a (slightly modified) preview of the simplified collector derivation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


{ pkgs, lib, ... }:
pkgs.buildGoModule rec {
 # (source, name, version, vendor hash omitted)

 subPackages = "collector/cmd/collector-metrics/collector-metrics.go";

 buildInputs = with pkgs; [ makeWrapper ];

 CGO_ENABLED = 0;
 ldflags = [ "-extldflags=-static" ];
 tags = [ "netgo" "static" ];

 installPhase = ''
 mkdir -p $out/bin
 cp $GOPATH/bin/collector-metrics $out/bin/scrutiny-collector-metrics
 wrapProgram $out/bin/scrutiny-collector-metrics \
 --prefix PATH : ${lib.makeBinPath [ pkgs.smartmontools ]}
 '';

 # (meta block omitted)
}

Contributing To nixpkgs #

With these changes made, it was time to open a pull request. The NixOS community provide some contributing guidelines which are worth a read before you start.

There was a structural transition in the pkgs directory lately, so this was my first contribution under that new structure. There was a detailed talk about that transition at Nixcon if you’d like more information.

I needed to do the following:

Create the scrutiny and scrutiny-collector packages
Add the scrutiny module
Add the tests
Update the release notes for the next release of NixOS

Which translated into the following stack of commits on PR #289934:

There were a few other minor changes, such as how input packages (like smartmontools and makeWrapper) are passed to the packages. In the nixpkgs repository, packages are passed directly to the derivations, rather than pkgs being passed as a top-level argument and packages being referred to by pkgs.smartmontools etc.

I also learned something new about module configuration. In the first iteration I had set the default value of services.scrutiny.collector.enable to the value of services.scrutiny.enable, meaning that the collector would be enabled automatically when Scrutiny was enabled if no other configuration was specified. It transpires that this is not permitted in nixpkgs, and resulted in this CI failure. I updated the module with a subtle API change, meaning that consumers of the module will now need to do the following to get the same behaviour.

1
2
3
4
5
6


{
 services = {
 scrutiny.enable = true;
 scrutiny.collector.enable = true;
 };
}

Review Feedback #

From the first review I learned about the lib.types.nullOr meta-type which enables the specification of optionally null configuration options in a NixOS module.

What followed was a suggestion to embrace a different configuration format according to RFC42: a proposal for structured module configuration which is more rigorously type checked, and more flexible as new options are added to the underlying workload.

Many NixOS modules solve this problem today with options like extraConfig or configLines, which append strings to the end of any options-generated configuration file. What RFC42 proposes is a hybrid approach whereby certain options are well-defined (and thus feature in the documentation), but additional options can be injected as needed. The has the nice side-effect that modules do not need to support dozens of options which can become a burden to maintain and test over time, but consumers can still make use of new features and options that are introduced to the underlying workload.

You can see the full resulting options definition on Github, and small annotated excerpt below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


{
 services.scrutiny = {
 enable = lib.mkEnableOption "Enables the scrutiny web application.";

 # ...

 settings = lib.mkOption {
 description = lib.mdDoc ''
 Scrutiny settings to be rendered into the configuration file.

 See https://github.com/AnalogJ/scrutiny/blob/master/example.scrutiny.yaml.
 '';

 default = { };

 # This is the key to RFC42 - defining `settings` as type `submodule`.
 type = lib.types.submodule {
 # This ensures that attrsets, lists, etc. are rendered as JSON
 freeformType = (pkgs.formats.yaml { }).type;

 # The following are examples of "well-defined" options with
 # formal types and metadata.
 options.web.listen.port = lib.mkOption {
 type = lib.types.port;
 default = 8080;
 description = lib.mdDoc "Port for web application to listen on.";
 };

 options.web.listen.host = lib.mkOption {
 type = lib.types.str;
 default = "0.0.0.0";
 description = lib.mdDoc "Interface address for web application to bind to.";
 };

 # ...
 };
 };
}

This enables the following user configurations:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


{
 services.scrutiny = {
 enable = true;
 settings = {
 # These options are well-defined in the module with types,
 # examples, descriptions and are rendered in the docs.
 web.listen = {
 port = 8080;
 host = "100.64.12.34";
 };

 # These are not, but will be rendered still as JSON and enable
 # people to make use of options supported by Scrutiny, but
 # not directly by the module.
 notify = {
 urls = [
 "discord://token@channel"
 ];
 };
 };
 };
}

I was impressed by the simplicity of the approach, and subsequently adopted the same for the collector, which now has the ability to take configuration through the services.scrutiny.collector.settings attribute.

One aspect of Scrutiny’s design that helped is the ability to take configuration either through from file, or through environment variables, or both! I chose to supply some fixed options as environment variables to the systemd services, and allow the rest of the configuration to be set using the generated file.

@JulienMalka reviewed next, suggesting changes to ensure that various preInstall and postInstall hooks were being called where I was overriding installPhase, though what followed from @katexochen’s review was a simplification that reduced this need slightly.

@katexochen suggested a couple of nice changes, among which was the ability to drop explicitly mentioning the netgo tag in the Go packages, since CGO_ENABLED is disabled already, this implies using the Go internal version of many libraries. They also suggested moving the install of the frontend pieces to postInstall, rather than overriding the standard installPhase.

And finally @SuperSandro2000 reviewed with some naming changes to simplify the package definitions and reduce repetition. I also noticed that I was unnecessarily declaring the frontend definition using the rec keyword. Once these were fixed the PR was merged!

Contribution Tips #

I’m still relatively new to this - you can see that my list of contributions is not extensive, but I’ve been through the process of adding, updating and refining packages and modules a few times.

Take this advice for what it is, but I hope you find it helpful:

Read the contribution guidelines: This is probably the best start. Try to follow the guidance as best you can, and take a look at some other recently closed Pull Requests to see if you can pick up on any patterns.
Be humble: Everyone is new when they start. You might find some reviews a little terse. Rest assured that this is only because the maintainers are incredibly busy. Assume good intentions, remain polite and respectful. Nixpkgs is one of the busiest Github repositories I’ve seen, and it’s largely maintained by volunteers who often have limited time to give for a huge number of contributions.
Be patient: Don’t expect a review immediately. If you’re looking for a preliminary review, feel free to ping me on Mastodon and I’ll try to offer any guidance I can.
Use the Discourse: Because of the volume of PRs, the community run ongoing “PRs Ready for Review” threads on Discourse which you can use to attract reviewers if your PR has been stale a while. It’s also a great place to ask for help.

Once merged, commits trickle through the various branches of the repository. You can track the progress of your PR in that process once it’s merged using the PR Tracker. You can see an example of this by tracking the Scrutiny PR:

Summary & Thanks #

My Scrutiny packages, module and tests landed in nixpkgs approximately 4 days after I submitted the pull request. As ever, I received a bunch of helpful reviews from the community, and Scrutiny is now easier than ever to consume on NixOS.

You can see the finished components here:

If you were previously using my flake to consume Scrutiny, you’ll now see a deprecation warning. To migrate over, import the module from unstable in your flake and ensure that you explicitly enable the collector:

1
2
3
4
5
6


{
 services = {
 scrutiny.enable = true;
 scrutiny.collector.enable = true;
 };
}

Before I wrap up, I’d like to thank all those people who took the time to review the PR, and especially @JulienMalka and @RaitoBezarius who have helped me through countless contributions over the past months and been incredibly responsive to my questions.

Give it a go and let me know how you get on!

Packaging Scrutiny for NixOS

Fri, 16 Feb 2024 00:00:00 +0000

Update: Since writing this post, I’ve contributed Scrutiny upstream to nixpkgs/NixOS. You can see the write up of that effort here.

Introduction #

In a recent (well, recent-ish) episode of the Self Hosted Show, there was some talk of a hard drive monitoring tool called Scrutiny. Scrutiny is a hard drive monitoring tool that exposes S.M.A.R.T data in a nice, clean dashboard. It gathers that S.M.A.R.T data using the venerable smartd, which is a Linux daemon that monitors S.M.A.R.T data from a huge number of ATA, IDE, SCSI-3 drives. The code is available on Github.

The aim of running such monitoring is to detect and replace failing hard drives before they cause an outage, or any data loss. Depending on the firmware of a drive, there are potentially hundreds of S.M.A.R.T attributes that can be collected, so it can be hard to understand which to pay attention to.

As good as smartd is, it has some shortcomings such as not recording historical data for each attribute, and relying upon value thresholds that are set by the manufacturers of the drive. Scrutiny aims to provide historical tracking of attribute readings, and thresholds based on real-world data from Backblaze.

This all looked very compelling, but it hasn’t been packaged for NixOS as far as I can tell. This post is quite “code heavy” and is intended as a bit of a deep-dive/walkthrough for people who are new to packaging in Nix. I’m certainly no expert here, and if you spot something I’ve done wrong - I’d love to hear about it!

Below is a screenshot of Scrutiny’s dashboard:

Architecture #

In order to get Scrutiny up and running, there were a few components that all needed packaging separately.

Scrutiny itself is made up of the following:

A web application backend - written in Go
A web application frontend - written in NodeJs/Angular
A collector service - written in Go

The web application is the dashboard that you’ll see the pretty screenshots of. The collector service is designed to be run on an interval to collect information from smartd, and send it to the web application’s API.

Scrutiny relies upon InfluxDB to store data about smart attributes in a timeseries.

Thinking about how to structure things, I decided that I would build two separate Nix packages: one for the dashboard and UI, and another for the collector. It seems one could run the collector and dashboard on different machines, so this seemed like a logical split.

Packaging for NixOS #

Packaging the Dashboard #

I decided to start with the web application. Because of how the project is laid out, this means combining two separate derivations based built from the same source code: one for the UI and one for the backend. I began by creating a file that would carry common attributes for each of the derivations, such as the version, repository information, hash, etc.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


# common.nix
{ pkgs , ... }:
let
 name = "scrutiny";
 version = "0.7.2";
in
{
 inherit name version;

 # Fetch the source code from Github, referring to
 # the version required by Git tag.
 repo = pkgs.fetchFromGitHub {
 owner = "AnalogJ";
 repo = name;
 rev = "v${version}";
 sha256 = "sha256-UYKi+WTsasUaE6irzMAHr66k7wXyec8FXc8AWjEk0qs=";
 };

 # ...
}

I then started packaging the frontend:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36


{ pkgs, ... }: {
 # Import some common attributes such
 # as name, version, src repo
 common = import ./common.nix { inherit pkgs; };
 inherit (common) name version repo vendorHash;

 # Create a package for the Javascript frontend
 frontend = pkgs.buildNpmPackage rec {
 inherit version;
 pname = "${name}-webapp-frontend";
 src = "${repo}/webapp/frontend";

 # This hash is generated at build time, and uniquely identifies
 # the cache of NodeJS dependencies used for the build.
 npmDepsHash = "sha256-M8P41LPg7oJ/C9abDuNM5Mn+OO0zK56CKi2BwLxv8oQ=";

 # Override the build phase to match the
 # upstream build process.
 buildPhase = ''
 runHook preBuild
 mkdir dist
 npm run build:prod --offline -- --output-path=dist
 runHook postBuild
 '';

 # Copy the output of the compiled javascript bundle
 # and site assets to the package output.
 installPhase = ''
 runHook preInstall
 mkdir $out
 cp -r dist/* $out
 runHook postInstall
 '';
 };
 # ...
}

This is a relatively simple derivation, mostly thanks to the magic of the buildNpmPackage function. This helper function takes care of creating an offline cache containing all of the NodeJS dependencies required to build the frontend. Nix builds are always done in an offline environment to help ensure reproducibility, so source code and dependencies are fetched (and hashed) early in the process before any software is actually built.

I chose to override the build phase to match the process used by the upstream Makefile. The result of this derivation is a package named scrutiny-webapp-frontend, which contains just the built output from the npm run build:prod command.

Next up was the dashboard backend. Another pleasingly simple derivation thanks to some help from buildGoModule:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


pkgs.buildGoModule rec {
 # The vendor hash is used for multiple packages, and
 # thus centralised and imported here.
 inherit version vendorHash;
 pname = "${name}-webapp-backend";

 # Use the source block defined in 'common.nix'
 src = repo;

 CGO_ENABLED = 0;

 # Override the build phase to ensure the correct
 # binary is built. This project ships both Go applications
 # and the NodeJS frontend in the same repository, so some
 # specificity is required.
 buildPhase = ''
 runHook preBuild
 go build \
 -o scrutiny-web \
 -ldflags="-extldflags=-static" \
 -tags "static netgo" \
 ./webapp/backend/cmd/scrutiny
 runHook postBuild
 '';

 # Add the built binary to the package output, as well
 # as the build output from the frontend.
 installPhase = ''
 mkdir -p $out/bin $out/share/scrutiny
 cp scrutiny-web $out/bin/scrutiny-web
 cp -r ${frontend}/* $out/share/scrutiny
 '';
}

This one is a little more interesting. There are some commonalities, such as setting the vendorHash to ensure the correct Go dependencies are used (imported from common.nix in this case), and overriding the build to match the upstream process and ensure the right binary is built.

Where things differ is in the install phase, where the contents of the scrutiny-webapp-frontend derivation is copied into the output of this derivation - which will ultimately result in a single Nix package (named scrutiny-webapp-backend) which will contain both the frontend and backend components of the application. This is ultimately exposed in an overlay as a package simply named scrutiny.

You can see the finished product (with additional package metadata) in app.nix and common.nix on Github.

Packaging the Collector #

The collector is just a single, statically compiled Go binary, and as such the derivation very much resembles that of the web application backend above:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39


{ pkgs, lib, ... }:
let
 # Again import common attributes from 'common.nix'
 common = import ./common.nix { inherit pkgs; };
 inherit (common) name version repo vendorHash;
in
pkgs.buildGoModule rec {
 inherit version vendorHash;
 pname = "${name}-collector";
 src = repo;

 # We'll create a wrapper script for the
 # collector, which makeWrapper helps with.
 buildInputs = with pkgs; [ makeWrapper ];

 CGO_ENABLED = 0;

 # Override the build phase to ensure the correct
 # binary is built.
 buildPhase = ''
 runHook preBuild
 go build \
 -o scrutiny-collector-metrics \
 -ldflags="-extldflags=-static" \
 -tags "static netgo" \
 ./collector/cmd/collector-metrics
 runHook postBuild
 '';

 # Install both the binary, and a generated wrapper script
 # which ensures that 'smartctl' is in the PATH of the collector.
 installPhase = ''
 mkdir -p $out/bin
 cp scrutiny-collector-metrics $out/bin/scrutiny-collector-metrics

 wrapProgram $out/bin/scrutiny-collector-metrics \
 --prefix PATH : ${lib.makeBinPath [ pkgs.smartmontools ]}
 '';
}

Of interest here is the installPhase. The collector works by invoking smartctl to scrape information from smartd. Scrutiny itself expects that tool to be readily available in it’s PATH, and to accomplish that I used the makeWrapper package to create a wrapper script that ensures scrutiny-collector-metrics is executed with the PATH correctly set such that smartctl can be found.

The final derivation for the collector can be seen in collector.nix on Github.

Writing a NixOS Module #

While I now had functioning packages for all of Scrutiny’s components, for them to function correctly on a machine there are a few things that need to be in place:

The dashboard package must be installed and started
The collector collector package must be installed and started
InfluxDB must be installed and started
Scrutiny dashboard must be configured to speak to the host’s InfluxDB
smartd must be installed and running

This sort of challenge is exactly what the NixOS modules system aims to solve. Modules take a set of configuration attributes, and convert them into rendered system configuration in the form of systemd units, configuration files and more.

It’s tempting to try to support all possible configurations when writing a module, but I generally prefer to start small, and support only the configuration I need for my use-case. This keeps things easy to test (and leaves some fun for the future!). In this case, the module would be first installed on a server which hosts a set of services behind the Traefik reverse proxy. To work out what configuration I wanted to provide, I looked at the upstream’s example config file. The important things that stood out to me for consideration were:

The location of the web UI files to serve
The host/port of the InfluxDB instance
The “base path” - Scrutiny will be exposed at https://<tailscale node>/scrutiny

As mentioned before - each NixOS module consists of some options, and some rendered config. The options block for my Scrutiny web app looks like the below snippet. I’ve omitted comments this time, as I think the language is quite descriptive here:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48


{
 let
 cfg = config.services.scrutiny;
 in
 options = {
 services.scrutiny = {
 enable = lib.mkEnableOption "Enables the scrutiny web application";

 # Use the `scrutiny` package by default.
 package = lib.mkPackageOptionMD pkgs "scrutiny" { };

 port = lib.mkOption {
 type = lib.types.port;
 default = 8080;
 description = lib.mdDoc "Port for web application to listen on";
 };

 host = lib.mkOption {
 type = lib.types.str;
 default = "0.0.0.0";
 description = lib.mdDoc "Interface address for web application to bind to";
 };

 basepath = lib.mkOption {
 type = lib.types.str;
 default = "";
 description = lib.mdDoc ''
 If Scrutiny will be behind a path prefixed reverse proxy, you can override this
 value to serve Scrutiny on a subpath.

 Do not include the leading '/'.
 '';
 };

 openFirewall = lib.mkOption {
 type = lib.types.bool;
 default = false;
 description = "Open the default ports in the firewall for Scrutiny.";
 };

 logLevel = lib.mkOption {
 type = lib.types.enum [ "INFO" "DEBUG" ];
 default = "INFO";
 description = lib.mdDoc "Log level for Scrutiny.";
 };
 };
 };
}

This provides some basic configuration for the attributes I care about - noticeably missing is any advanced configuration for InfluxDB (such as org, bucket, token), and and any ability to configure notifications which Scrutiny supports through the excellent shoutrrr library. These things will come later.

I needed a convenient way to convert this Nix-native configuration format into the right format for Scrutiny - I wrote a small Nix function to help with that, which takes configuration elements from the options defined above, and writes them into a small YAML file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


{
 mkScrutinyCfg = cfg: pkgs.writeTextFile {
 name = "scrutiny.yaml";
 text = ''
 version: 1
 web:
 listen:
 port: ${builtins.toString cfg.port}
 host: "${cfg.host}"
 basepath: "${if cfg.basepath != "" then "/" + cfg.basepath else ""}"

 database:
 location: "/var/lib/scrutiny/scrutiny.db"

 src:
 frontend:
 path: "${pkgs.scrutiny}/share/scrutiny"

 log:
 level: "${cfg.logLevel}"
 '';
 };
}

Note that this hard-codes some key elements, such as the path of the web application assets which are shipped as part of the scrutiny package.

Let’s take a look at the part of the module which turns these options into a rendered system configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


{
 config = {
 # If scrutiny is enabled, also enable InfluxDB with default settings
 services.influxdb2 = lib.mkIf cfg.enable {
 enable = true;
 };

 # Open the relevant ports in the system firewall if configured
 networking.firewall = lib.mkIf cfg.openFirewall {
 allowedTCPPorts = [ cfg.port ];
 };

 # If Scrutiny is enabled, create a systemd unit to start it
 systemd = {
 services = {
 scrutiny = lib.mkIf cfg.enable {
 description = "Hard Drive S.M.A.R.T Monitoring, Historical Trends & Real World Failure Thresholds";
 wantedBy = [ "multi-user.target" ];
 after = [ "network.target" ];
 serviceConfig = {
 # Don't run the application a root - create a dynamic user as it starts
 DynamicUser = true;
 # Start the application with a config rendered by the helper function
 ExecStart = "${cfg.package}/bin/scrutiny-web start --config ${mkScrutinyCfg cfg}";
 Restart = "always";
 StateDirectory = "scrutiny";
 StateDirectoryMode = "0750";
 };
 };
 };
 };
 };
}

That’s enough to get the dashboard started, but it doesn’t take care of starting the collector. For that, I added a couple more configuration options to the module:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


{
 let
 collectorCfg = config.services.scrutiny.collector;
 in
 {
 options = {
 services.scrutiny = {
 # ...
 collector = {
 enable = lib.mkOption {
 type = lib.types.bool;
 default = cfg.enable;
 description = lib.mdDoc "Enables the scrutiny collector";
 };

 # Use the `scrutiny-collector` package by default.
 package = lib.mkPackageOptionMD pkgs "scrutiny-collector" { };

 endpoint = lib.mkOption {
 type = lib.types.str;
 default = "http://${cfg.host}:${builtins.toString cfg.port}/${cfg.basepath}";
 description = lib.mdDoc "Scrutiny app API endpoint for sending metrics to.";
 };

 interval = lib.mkOption {
 type = lib.types.str;
 default = "15m";
 description = lib.mdDoc ''
 Interval on which to collect information about disks.

 Examples: 15m, 10s, 2h.
 '';
 };
 };
 };
 };
 }
}

With that configuration in place, I needed to adjust the rendered configuration to include starting the collector. The collector is designed to run on an interval to post metrics to the dashboard. The upstream achieves this with cron, but I decided to use systemd timers:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39


{
 config = {
 # ...
 # If the collector is enabled, enable smartd so that the
 # collector can scrape metrics from it
 services.smartd = {
 enable = collectorCfg.enable;
 extraOptions = [ "-A /var/log/smartd/" "--interval=600" ];
 };

 systemd = {
 services = {
 # Setup a systemd unit to start the collector
 scrutiny-collector = lib.mkIf collectorCfg.enable {
 description = "Scrutiny Collector Service";
 environment = {
 COLLECTOR_API_ENDPOINT = "${collectorCfg.endpoint}";
 };
 serviceConfig = {
 Type = "oneshot";
 ExecStart = "${cfg.collector.package}/bin/scrutiny-collector-metrics run";
 };
 };
 };

 # Set up a systemd timer to trigger the collector at an
 # interval (default 15m).
 timers = {
 scrutiny-collector = lib.mkIf collectorCfg.enable {
 timerConfig = {
 OnBootSec = "5m";
 OnUnitActiveSec = "${collectorCfg.interval}";
 Unit = "scrutiny-collector.service";
 };
 };
 };
 };
 };
}

And that’s it! You can see the final module all stitched together on Github.

Automated Testing #

To me, a super exciting part of the NixOS ecosystem is its automated testing framework, which is used to great effect for testing hundreds of packages end-to-end before they’re released to the various NixOS channels. The NixOS testing framework provides a set of helpers for spawning fresh NixOS virtual machines, and ensuring that applications can start, services have the right side effects, etc.

I wanted to write a simple test to validate that Scrutiny will continue to work as I update my flake. In my view, the test needed to:

Ensure that the packages could be built
Ensure that when services.scrutiny.enable = true is set, the services start
Ensure that the dashboard app renders the UI
Ensure that the metrics collector can speak to the dashboard

Most of these are relatively trivial - one piece that’s a little harder is testing that the UI renders correctly. The application is rendered using Javascript client-side, so a simple curl won’t get us the results we’re expecting. I had previously used selenium for this purpose when I submitted a test for the LXD UI, so I chose to use that approach again.

The test code can be seen below, or on Github:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69


{
 name = "scrutiny";
 nodes = {
 # Configure a NixOS virtual machine for the test
 machine = { self, pkgs, lib, ... }: {
 # Ensure that my NixOS module is imported
 imports = [ self.nixosModules.scrutiny ];
 # Add the package overlay from my flake so that `pkgs.scrutiny` resolves.
 nixpkgs.overlays = [ self.outputs.overlays.additions ];

 # Configure the VM to use my module and enable it
 services.scrutiny.enable = true;


 environment.systemPackages =
 let
 # A selenium script that fetches the dashboard using geckodriver
 seleniumScript = pkgs.writers.writePython3Bin "selenium-script"
 {
 libraries = with pkgs.python3Packages; [ selenium ];
 } ''
 from selenium import webdriver
 from selenium.webdriver.common.by import By
 from selenium.webdriver.firefox.options import Options
 from selenium.webdriver.support.ui import WebDriverWait

 options = Options()
 options.add_argument("--headless")
 service = webdriver.FirefoxService(executable_path="${lib.getExe pkgs.geckodriver}") # noqa: E501

 driver = webdriver.Firefox(options=options, service=service)
 driver.implicitly_wait(10)
 driver.get("http://localhost:8080/web/dashboard")

 wait = WebDriverWait(driver, 60)

 # Look for some elements that should be rendered by the Javascript bundle in the UI
 assert len(driver.find_elements(By.CLASS_NAME, "mat-button-wrapper")) > 0
 assert len(driver.find_elements(By.CLASS_NAME, "top-bar")) > 0

 driver.close()
 '';
 in
 with pkgs; [ curl firefox-unwrapped geckodriver seleniumScript ];
 };
 };
 # This is the test code that will check if our service is running correctly:
 testScript = ''
 start_all()

 # Wait for InfluxDB to be available
 machine.wait_for_unit("influxdb2")
 machine.wait_for_open_port(8086)

 # Wait for Scrutiny to be available
 machine.wait_for_unit("scrutiny")
 machine.wait_for_open_port(8080)

 # Ensure the API responds as we expect
 output = machine.succeed("curl localhost:8080/api/health")
 assert output == '{"success":true}'

 # Start the collector service to send some metrics
 collect = machine.succeed("systemctl start scrutiny-collector.service")

 # Ensure the application is actually rendered by the Javascript
 machine.succeed("PYTHONUNBUFFERED=1 selenium-script")
 '';
}

This relatively short code snippet will take care of:

Building a dedicated virtual machine according to the machine spec
Starting the VM
Running the tests inside the VM (specified in `testScript)
Collecting the results

I ended up adding this check in a Github Actions workflow so that it’s run each time I make a change to my flake.

Try It! #

You can give this a go too! If you’ve got a NixOS machine, or perhaps a virtual machine you’ve been experimenting with, then you need only add the following to your flake:

1
2
3
4
5
6
7
8
9


{
 inputs = {
 nixpkgs-unstable.url = "github:nixos/nixpkgs/nixos-unstable";
 jnsgruk.url = "github:jnsgruk/nixos-config";
 jnsgruk.inputs.nixpkgs.follows = "nixpkgs-unstable";
 # ...
 }
 # ...
}

And in your NixOS machine configuration:

1
2
3
4
5
6


# Where 'inputs' is the inputs attribute of your flake
{ inputs, ... }: {
 imports = [ inputs.jnsgruk.nixosModules.scrutiny ];
 nixpkgs.overlays = [ inputs.jnsgruk.overlays.additions ];
 services.scrutiny.enable = true;
}

The next time you rebuild your system, you should be able to browse to http://localhost:8080 and see some hard drive metrics!

What’s Next? #

This was a pretty quick exercise - I’d estimate at about 3 hours in total. No doubt I will find some bugs, but already in my mind are the following future improvements:

Add some more tests that exercise more of the config options
Enable notifications configuration in the module
Submit the packages/module to the upstream or to nixpkgs

I also need to go and investigate this rather sad looking hard disk…

That’s all for now! Thanks for reading if you got this far!

Integration testing with NixOS in Github Actions

Sat, 10 Feb 2024 00:00:00 +0000

Introduction #

While in my own time I’ve tended toward NixOS over the past 18 months, in my day-to-day work for Canonical I’m required to interact with a fair few of our products - and particularly build tools.

I frequently need to use some combination of Snaps, Charms and Rocks. Each of these have their own “craft” build tools (snapcraft, charmcraft and rockcraft), which are distributed exclusively as Snap packages and thus a little tricky to consume from NixOS.

The Problem #

Packaging the tools for Nix was a little repetitive, but not particularly difficult. They’re all built with Python, and share a common set of libraries. Testing that the packages were working correctly (i.e. could actually build software) on NixOS using the NixOS version of LXD in Github Actions proved more difficult.

Github Actions defaults to Ubuntu as the operating system for its runners - an entirely sensible choice, but not one that was going to help me test packages could work together on NixOS.

I could have hosted my own Github Actions runners to solve the problem, but I didn’t want to maintain such a deployment.

For a while I relied on just testing each of the crafts locally before pushing, and the CI simply installed the Nix package manager on the runners (using the excellent Nix installer from Determinate Systems) and ensured that the build could succeed, but this left a lot to be desired - particularly when I accidentally (and somewhat inevitably) broke one of the packages.

KVM for Github Actions #

Some time later I came across this post on the Github Blog, stating the following:

Starting on February 23, 2023, Actions users […] will be able to make use of hardware acceleration […].

What follows is an example of a relatively simple addition to a Github Workflow to enable KVM on Github Actions runners:

1
2
3
4
5


- name: Enable KVM group perms
 run: |
 echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules
 sudo udevadm control --reload-rules
 sudo udevadm trigger --name-match=kvm

Given the ability to relatively easily create NixOS VMs from a machine configuration, this should enable me to run a NixOS VM inside my Github Actions runners, and use that VM to run end to end tests of my craft packages.

After some quick tests, I confirmed that the above snippet worked just fine on the freely available runners that are assigned to public projects. After tooting excitedly about this, it was also picked up by the folks at Determinate Systems who promptly added support for this in their Nix install Github Action - enabling the feature by default.

Building VMs with Nix #

A really nice feature of NixOS that I discovered relatively late, is that given a NixOS machine configuration it’s trivial to build a virtual machine image for that configuration. This has the nice property that one can actually boot a VM-equivalent of any previously defined machines. You could, for example, boot a VM-equivalent of my laptop with the following command:

1

nix run github:jnsgruk/nixos-config#nixosConfigurations.freyja.config.system.build.vm

In order to test my craft tools, I needed a relatively simple NixOS VM that had LXD enabled, and my craft tools installed. My test VM configuration looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45


{ modulesPath, flake, pkgs, ... }: {
 # A nice helper that handles creating the VM launch script, which in turn
 # ensures the disk image is created as required, and QEMU is launched
 # with sensible parameters.
 imports = [ "${modulesPath}/virtualisation/qemu-vm.nix" ];

 # Define the version of NixOS and the architecture.
 system.stateVersion = "23.11";
 nixpkgs.hostPlatform = "x86_64-linux";

 # This overlay is provided by the crafts-flake, and ensures that
 # 'pkgs.snapcraft', 'pkgs.charmcraft', 'pkgs.rockcraft' all resolve to
 # the packages in the flake.
 nixpkgs.overlays = [ flake.outputs.overlay ];

 # These values are tuned such that the VM performs on Github Actions runners.
 virtualisation = {
 forwardPorts = [{ from = "host"; host.port = 2222; guest.port = 22; }];
 cores = 2;
 memorySize = 5120;
 diskSize = 10240;
 };

 # Configure the root user without password and enable SSH.
 # This VM will only ever be used in short-lived testing environments with
 # no inbound networking permitted, so there is minimal (if any) risk.
 # If you put this VM on the internet, you can keep the pieces! :)
 networking.firewall.enable = false;
 services.openssh.enable = true;
 services.openssh.settings.PermitRootLogin = "yes";
 users.extraUsers.root.password = "password";

 # Ensure that LXD is installed, and started on boot.
 virtualisation.lxd.enable = true;

 # Include the `craft-test` script, ensuring the craft apps are installed
 # and included in its PATH.
 environment.systemPackages = [
 (pkgs.writeShellApplication {
 name = "craft-test";
 runtimeInputs = with pkgs; [ unixtools.xxd git snapcraft charmcraft rockcraft ];
 text = builtins.readFile ./craft-test;
 })
 ];
}

Anybody can build and launch this VM trivially:

1

nix run github:jnsgruk/crafts-flake#testVM

Writing a Github workflow #

All the building blocks are in place! I wanted to keep the actual workflow definition for the tests as clean and understandable as possible, so I put together the craft-test script as a small helper which automates the building of real artefacts. An example invocation might be:

1

bash craft-test snapcraft

On each invocation, the script creates temporary directory, clones some representative build files for the selected craft tool, and launches the craft. The repos it uses for the representative packages are hard-coded for each craft for now.

I wrote one more small helper script to simplify connecting to the VM with the required parameters. It’s a wrapper around ssh and sshpass that’s hard-coded with the credentials of the test VM (don’t @ me!), and executes commands over SSH in the test VM. Using this script, one can bash vm-exec -- craft-test snapcraft and the craft-test script will be executed over SSH in the VM.

With all that said and done, the resulting workflow is pleasingly simple:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


# ...
jobs:
 test:
 runs-on: ubuntu-latest
 strategy:
 matrix:
 package: ["charmcraft", "rockcraft", "snapcraft"]
 steps:
 - name: Checkout flake
 uses: actions/checkout@v4

 - name: Install nix
 uses: DeterminateSystems/nix-installer-action@v9

 - name: Build and run the test VM
 run: |
 nix run .#testVm -- -daemonize -display none

 - name: Test ${{ matrix.package }}
 run: |
 nix run .#testVmExec -- craft-test ${{ matrix.package }}

A separate job is run for each of the crafts, and a real artefact is built in each, giving reasonable confidence that the consumers of my flake will be successful when building snaps, rocks and charms natively on NixOS. A successful run can be seen here.

Summary #

In this article we’ve covered:

Enabling KVM on Github Runners
Building NixOS VMs using Flakes
Booting NixOS VMs in Github Actions

If you’d like to build snaps, rocks or charms and you’re running NixOS, you can run the tools individually from my flake:

1
2
3
4
5
6
7
8


# Run charmcraft
nix run github:jnsgruk/crafts-flake#charmcraft

# Run rockcraft
nix run github:jnsgruk/crafts-flake#rockcraft

# Run snapcraft
nix run github:jnsgruk/crafts-flake#snapcraft

Or you can check out the README for instructions on how to integrate into your Nix config using overlays!

That’s all for now! 🤓

Building a blog with Go, Nix and Hugo

Fri, 12 Jan 2024 00:00:00 +0000

Introduction #

I’ve been procrastinating about blogging for about a decade. Long-form writing is a format I enjoy consuming, and I’ve learned a huge amount from the various blogs I’ve subscribed to over the years. Yet, there have always been a couple of nagging reasons preventing me from starting my own:

Why would anyone want to read my blog?
How would I come up with ideas for content?
Where would I find the time to write the blog?

Perhaps what’s changed recently is a new found enjoyment in some side projects, and the realisation that I might just write about things for the love of it, whether or not its directly useful to anyone else. Of course I’d love people find the content useful, engage, etc., but that isn’t my primary motivation.

The second two points are closely linked, but I ultimately decided they didn’t matter. So I present this blog as a self-indulgence, and something that I’ll update when I’m excited about writing, and not feel bad about the rest of the time! 😉

Being a Software Engineer, I quickly established that it was important to spend time over-engineering my blog before sitting down and writing any content, and this first post illustrates that journey </sarcasm>.

Rendering the blog #

I’ve been fond of Hugo for years now (I even named my son Hugo 😉). I’ve used it in a few projects, and I find it to be largely easy to understand and well maintained. My previous site was built with Hugo, using a theme named congo which I’d been underutilising by only creating a “business card” style page. I decided to stick with this setup, and just use more of the layouts provided by the theme.

In many ways the Hugo site is the most “boring” part of the site. The source code is all available in the site directory of the Github repo, but I won’t talk much about the site itself in this blog, as there isn’t much more to say!

Serving the blog #

4 years ago, I made the first commit to a project named gosherve. This was one of my first adventures into Go, and I was left with a small, but functional web server that could serve files from a directory, and serve redirects specified in a publicly accessible text file.

I chose to host the redirect definitions in a Github Gist. When I want to share a link frequently, or place one somewhere visible like a slide, I update the Gist with a new alias, and jnsgr.uk/<alias> comes online as a handy short link the first time someone requests it.

Last year I decided to use gosherve as a tool for learning more about slog and the Go Prometheus client. I did some refactoring that tidied up the logging, and introduced basic metrics for the number of times each redirect was accessed, how many redirects were defined and the total number of redirects served.

For my new blog, I wanted to keep the short URLs I’d defined, and I wanted to embed the static site into the binary to make deployment as simple as possible. I made two small changes to gosherve to enable this:

The first change enables the server and logging components of gosherve to be imported as libraries, and the second enables gosherve to serve files from a filesystem (and critically, an embedded filesystem).

Embedding the blog #

One of the things I love about Go is how rich the standard library is, and how it can simplify the creation of small, but powerful applications. The code for my website’s server as I write this is below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40


package main

//go:generate hugo --minify -s site -d ../public

import (
 "embed"
 // ...
 "github.com/jnsgruk/gosherve/pkg/logging"
 "github.com/jnsgruk/gosherve/pkg/server"
)

var (
 commit string = "dev"
 logLevel = flag.String("log-level", "info", "log level of the application")
 redirectsURL = "https://gist.githubusercontent.com/jnsgruk/b590f114af1b041eeeab3e7f6e9851b7/raw"

 //go:embed public
 publicFS embed.FS
)

func main() {
 flag.Parse()
 logging.SetupLogger(*logLevel)

 fsys, err := fs.Sub(publicFS, "public")
 if err != nil {
 slog.Error(err.Error())
 os.Exit(1)
 }

 s := server.NewServer(&fsys, redirectsURL)

 err = s.RefreshRedirects()
 if err != nil {
 slog.Error("unable to fetch redirect map", "error", err.Error())
 os.Exit(1)
 }

 s.Start()
}

I’ve omitted some comments, imports and logging for brevity here, but the complete file (at 55 lines) can be found on Github for the curious.

There key elements here are:

//go:generate hugo --minify -s site -d ../public: this makes sure go generate invokes Hugo to build the site and place the output in the public directory.
//go:embed public: embeds the public directory into the binary as an embedded filesystem.

Building the blog #

For the last 18 months, I’ve been enjoying Nix and NixOS for my personal machines, so I wanted to use my newly acquired knowledge to package and build my website using Nix.

There is lots of ongoing discussion in the Nix community about Flakes, which are an experimental technology aimed at simplifying usability and improving reproducibility of Nix installations. There are lots of other facets to the discussion which I’ll likely touch upon in future posts, but for now I’ll just say that I like Flakes, and they were the obvious choice for packaging this site.

Packaging a Go application for Nix is relatively simple thanks to helpers like buildGoModule. I had to make some minor modifications to accommodate the go generate step to build the Hugo site, and patch out some elements of the Hugo site that relied upon access to the local Git tree, but the resulting derivation remains relatively easy to digest (see flake.nix):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34


buildGoModule {
 inherit version;
 pname = "jnsgruk";
 src = lib.cleanSource ./.;

 vendorHash = "sha256-4f04IS76JtH+I4Xpu6gF8JQSO3TM7p56mCs8BwyPo8U=";
 buildInputs = [ cacert ];
 nativeBuildInputs = [ hugo ];

 # Nix doesn't play well with Hugo's "GitInfo" module, so disable it and inject
 # the revision from the flake.
 postPatch = ''
 substituteInPlace ./site/layouts/shortcodes/gitinfo.html \
 --replace "{{ .Page.GitInfo.Hash }}" "${rev}"

 substituteInPlace ./site/config/_default/config.yaml \
 --replace "enableGitInfo: true" "enableGitInfo: false"
 '';

 # Generate the Hugo site before building the Go application which embeds the
 # built site.
 preBuild = ''
 go generate ./...
 '';

 ldflags = [ "-X main.commit=${rev}" ];

 # Rename the main executable in the output directory
 postInstall = ''
 mv $out/bin/jnsgr.uk $out/bin/jnsgruk
 '';

 meta.mainProgram = "jnsgruk";
};

This defines a Nix package named jnsgruk, containing a single binary at bin/jnsgruk. This binary can be run anywhere to get a working version of this site. You can even try at home with:

1

nix run github:jnsgruk/jnsgr.uk

Deploying the blog #

I’ve been hosting my site on Fly.io without issue for a couple of years. They have a nice feature that allows you to Deploy via Dockerfile, where their command-line utility flyctl will send off a local Dockerfile to be built on their infrastructure and then launched, and that had been working great in previous versions of my site.

I wanted to be able to build and run the exact same bits on my own machines as were hosted by Fly, so I opted to build an OCI image with Nix, then upload that to Fly.io’s registry as part of the deployment. Adding a container image to the flake as an additional output was simple:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


dockerTools.buildImage {
 name = "jnsgruk/jnsgr.uk";
 tag = version;
 created = "now";
 copyToRoot = buildEnv {
 name = "image-root";
 paths = [ self.packages.${system}.jnsgruk cacert ];
 pathsToLink = [ "/bin" "/etc/ssl/certs" ];
 };
 config = {
 Entrypoint = [ "${lib.getExe self.packages.${system}.jnsgruk}" ];
 Expose = [ 8080 8801 ];
 User = "10000:10000";
 };
};

All that remained was to wire up Github Actions to build and deploy the site each time I make a new commit. Because the build tooling setup is all handled by Nix, the resulting Github workflow is quite brief:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41


name: Fly Deploy
on:
 push:
 branches:
 - main

permissions:
 packages: write

jobs:
 deploy:
 name: Deploy app
 runs-on: ubuntu-latest
 steps:
 - name: Checkout
 uses: actions/checkout@v4

 - name: Install nix
 uses: DeterminateSystems/nix-installer-action@v9

 - name: Login to GitHub Container Registry
 uses: docker/login-action@v3
 with:
 registry: ghcr.io
 username: ${{ github.actor }}
 password: ${{ secrets.GITHUB_TOKEN }}

 - name: Build container
 run: nix build -L .#jnsgruk-container

 - name: Upload container to ghcr.io
 run: |
 docker load < result
 docker tag "jnsgruk/jnsgr.uk:$(git rev-parse --short HEAD)" "ghcr.io/jnsgruk/jnsgr.uk:$(git rev-parse --short HEAD)"
 docker push "ghcr.io/jnsgruk/jnsgr.uk:$(git rev-parse --short HEAD)"

 - name: Deploy site
 run: |
 nix run nixpkgs#flyctl -- deploy -i "ghcr.io/jnsgruk/jnsgr.uk:$(git rev-parse --short HEAD)"
 env:
 FLY_ACCESS_TOKEN: ${{ secrets.FLY_API_TOKEN }}

And that’s the end! You’re reading this article as a result of the above workflow succeeding.

Summary #

I’ve never had a blog before, but I’m looking forward to documenting some of my adventures in Linux, Software Engineering, Technical Leadership and more over the coming year. Thanks for reading!

Blog on Jon Seager

ntpd-rs: it's about time!

Introduction #

NTP, NTS, and PTP #

Proven at Scale #

A Single, Memory-Safe Utility for NTP and PTP #

Timelines and Goals #

About the Trifecta Tech Foundation #

Summary #

Brewlog: Coffee & Agents

Introduction #

Core Features #

LLM-Powered Bag Scanning #

Design Decisions #

Backend #

Frontend #

Single User, Passkeys Only #

Comprehensive CLI Client #

Domain-Driven Design #

Agentic Coding Patterns #

Pre-commit hooks as a contract #

Stricter lints for agents #

Self-updating instructions #

Plan Mode #

Worktrees and Parallel Agents #

Visual prompting #

Summary #

An update on upki

Architecture & Progress #

How to try it #

Next Steps #

Further Down the Road #

Certificate Transparency Enforcement #

Intermediate Preloading #

Merkle Tree Certificates #

Summary #

Developing with AI on Ubuntu

Hardware & Drivers #

Inference Snaps #

Sandboxing Agents #

My agent sandboxes itself! #

Sandbox with LXD containers #

Sandbox with LXD VMs #

Sandbox with Multipass #

Sandbox with WSL #

Summary #

Addressing Linux's Missing PKI Infrastructure

The Problem #

CRLite #

Introducing upki #

Ecosystem Compatibility #

Timeline #

Summary #

Ubuntu Summit 25.10: Personal Highlights

DOOM in Space #

Infrastructure-Wide Profiling of Nvidia CUDA #

Inference Snaps #

Nøughty Linux: Ubuntu’s Stability Meets Nixpkgs’ Freshness #

Are we stuck with the same Desktop UX forever? #

Honorable Mentions #

Conclusion #

Ubuntu Engineering in 2025: A Retrospective

Ubuntu 25.10: A Retrospective #

Communication #

Automation #

Process #

Modernisation #

What’s Next? #

The Immutable Linux Paradox

What is an immutable Linux distribution? #

Why immutability? #

The immutability paradox #

Approaches to immutability #

Fedora Silverblue / CoreOS / EndlessOS (ostree) #

RHEL “Image Mode” (bootc) #

NixOS #

Ubuntu Core #

Summary #

Crafting Your Software

Software build lifecycle #

Fedora Silverblue / CoreOS / EndlessOS (`ostree`) #

RHEL “Image Mode” (`bootc`) #

What is `sudo-rs`? #

Progress on `coreutils` #

Introducing `oxidizr` #