Development on Jon Seager

Brewlog: Coffee & Agents

Tue, 10 Mar 2026 00:00:00 +0000

Introduction #

I’m really into speciality coffee. In July 2025 I started tracking my coffee habits with Roastguide, a delightfully designed iOS app for people with a similar obsession to mine. It does a good job of tracking brews, roasters and bags, but over time I found myself wanting the data in a system that I controlled. I wanted to be able to query it, back it up, and adjust some of the flows to work better for how I brew and consume coffee.

Despite my past scepticism of the previous generation of coding tools, recent developments have made them hard to ignore, and I wanted an excuse to build something substantial from scratch to get some experience.

So I built b{rew}log, which is a self-hosted coffee logging platform. It tracks roasters, roasts, bags, brews, equipment, and cafe visits. It’s live at coffee.jnsgr.uk if you want to poke around and witness the depths of my strange filter coffee obsession!

This was by far the most complex project I’d built with Claude Code, or agentic coding tools in general. I’m responsible for the technology choices, architecture, and visual design, but I wrote almost none of the code myself.

This post will cover the core features of Brewlog and how I designed the app, and finish with some observations and tips about agentic programming.

Core Features #

Brewlog tracks roasters, roasts, bags, brews, equipment, and cafe visits. The landing page you see below summarises details of my last few brews, currently open bags of coffee and how much remains in each and some basic stats. When logged in, it provides quick controls to repeat a brew, or brew a particular coffee:

Brews are coffees I’ve made myself. Each brew is logged against a specific bag and captures the recipe (dose, water, time, temperature), the equipment used, and tasting notes. Over time this will build up a history of what I’ve brewed, how I brewed it and how much I use the brewing gear I’ve accumulated over time. The image below shows the detail page for a specific brew:

Finally, brewlog tracks cafes I’ve visited, and “cups”, which are coffees I’ve enjoyed, but not brewed myself.

The most fun part is the extensive stats page, which shows an interactive map of where my coffees are grown, roasted and drunk, as well as common flavour notes and brew times.

LLM-Powered Bag Scanning #

Roastguide has a nice feature that allows you to take a picture of a bag of coffee and it’ll fetch the details. If my understanding is correct, their implementation relies on a database of roasters and coffees that the creators maintain. My app wouldn’t have access to that database, nor did I want to spend too much time building ingestion pipelines to store lots of data about all the possible coffees/roasters.

In brewlog, I implemented this feature using LLM extraction: I take a photo and send it to a multi-modal model hosted by OpenRouter. The prompt instructs the model to extract text from the image, perform a web search, and return a JSON object containing details of the coffee or roaster. OpenRouter enables me to switch models easily without worrying about API differences.

In practice this works surprisingly well. Most speciality coffee bags are covered in useful information, and vision models are good at reading it. The main failure modes I see are mixing up roast names from the web and occasionally inventing tasting notes, but the process includes a review step which makes those cheap to correct.

The cost is almost negligible per scan. I’ve been using google/gemini-2.5-flash for the LLM extraction feature, which results in a cost of around $0.01 - $0.02 per scan.

Design Decisions #

Backend #

Brewlog is built with Rust and Axum for the backend. I’ve been learning Rust for a while now, and wanted to push further with a more substantial project. Templates are handled by Askama, whose compile‑time templates worked well with Claude because template errors surface as Rust compiler errors, which are picked up and fixed by the agent automatically.

The database is SQLite. A single file, easy to back up, easy to move around. For a single-user application like this, SQLite is more than sufficient and removes the need for a separate database server. Migrations are embedded in the binary and run automatically on application startup.

Frontend #

I wanted the app to feel modern, but keep much of the rendering server-side. I didn’t want to be serving a large client-side Javascript single-page application. I’ve been curious about the likes of HTMX, and the observations made in a post from last year on building apps with Eta, HTMX and Lit really resonated with me.

More recently I discovered Datastar, which has similar goals to HTMX, but it’s newer and results in cleaner, simpler templates. Where HTMX swaps HTML fragments, Datastar adds a reactive signal system and can patch both HTML and JSON data from the server.

This was a bit of a test for Claude Code, since Datastar was new enough that it likely didn’t feature in the training data for the model, but its ability to read and digest the documentation was quite startling.

While occasionally the agent regressed to vanilla fetch calls, or manual Javascript DOM manipulation, I treated that as a hole in my instructions, not the model. Each time it happened I updated CLAUDE.md to reinforce the Datastar patterns and prevent the agent making the same mistake again.

Single User, Passkeys Only #

Brewlog is deliberately single-user. I’m happy for people to self-host their own instance, but I don’t have much interest in providing a service more widely. The codebase is structured such that making it multi-user wouldn’t be a particularly complex task, but I’m not convinced I’ll ever do it.

Perhaps unusually, authentication via username and password is not supported. Authentication is only supported using passkeys via WebAuthn or with an API token.

Passkeys are phishing-resistant and significantly more difficult to brute-force both from an academic and a practical standpoint, which removes an entire class of abuse vector for my service.

Comprehensive CLI Client #

One of the many things I’ve learned from @niemeyer during my time at Canonical is to ensure that when new API endpoints are added to such applications, a corresponding CLI command or flag lands at the same time.

He also illustrated to me the power of shipping a single binary that is both a server, and a client to itself. The brewlog binary runs the server that hosts the API and web UI, but also serves as a first-class API client for the app.

If you’re building an API‑backed app, I highly recommend this pattern. It keeps the surface area small and gives you (and an agent…) a first‑class, scriptable way to exercise the API and troubleshoot problems in data transformation.

Domain-Driven Design #

The codebase follows a domain-driven design approach with four layers: domain (business logic only, no external dependencies), infrastructure (database, HTTP clients, third-party APIs), application (HTTP server, routes, middleware, services), and presentation (CLI commands and web view models):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36


┌─────────────────────────────────────────────────────────────┐
│ Presentation │
│ ┌─────────────────────────┐ ┌──────────────────────────┐ │
│ │ CLI │ │ Web │ │
│ │ roasters, roasts, bags │ │ views, templates │ │
│ │ brews, cups, gear ... │ │ roasters, roasts, bags │ │
│ └─────────────────────────┘ └──────────────────────────┘ │
├─────────────────────────────────────────────────────────────┤
│ Application │
│ ┌──────────────┐ ┌──────────────────┐ ┌─────────────────┐ │
│ │ Routes │ │ Services │ │ Server / State │ │
│ │ /api/* │ │ BagService │ │ Axum router │ │
│ │ /app/* │ │ BrewService │ │ AppState (DI) │ │
│ │ │ │ RoastService .. │ │ │ │
│ └──────────────┘ └──────────────────┘ └─────────────────┘ │
├─────────────────────────────────────────────────────────────┤
│ Domain (pure Rust — no framework dependencies) │
│ ┌──────────────────────┐ ┌─────────────────────────────┐ │
│ │ Entities & Values │ │ Repository Traits │ │
│ │ coffee/ roasters, │ │ trait RoasterRepository │ │
│ │ roasts, bags, │ │ trait RoastRepository │ │
│ │ brews, cups, gear │ │ trait BagRepository ... │ │
│ │ auth/ users, │ │ │ │
│ │ sessions, tokens │ │ Errors, IDs, Listing, │ │
│ │ analytics/ │ │ Formatting │ │
│ └──────────────────────┘ └─────────────────────────────┘ │
├──────────────────────────────────┬──────────────────────────┤
│ Infrastructure │ │
│ ┌────────────────────────────┐ │ ┌───────────────────┐ │
│ │ Repositories (SQLite) │ │ │ External Clients │ │
│ │ SqlRoasterRepository │ │ │ OpenRouter (AI) │ │
│ │ SqlRoastRepository │ │ │ Foursquare │ │
│ │ SqlBagRepository ... │ │ │ WebAuthn │ │
│ │ (implement domain traits) │ │ │ Backup │ │
│ └────────────────────────────┘ │ └───────────────────┘ │
└──────────────────────────────────┴──────────────────────────┘

In this model, dependencies flow inward. The domain layer knows nothing about Axum, SQLite, or any other implementation detail. Repository traits are defined in the domain and implemented in the infrastructure layer. This creates flexibility in the future if, for example, I want to move to PostgreSQL rather than SQLite, or if I want to change how I store images, etc.

It also simplifies testing. The domain-driven design approach encourages loose coupling and practices like dependency injection, which usually lead to simpler integration tests. For example, I could ask the agent for focused tests against pure domain logic without pulling in Axum or SQLite.

Agentic Coding Patterns #

A big motivation for this project was to learn some new technology and skills, and it certainly did that, but it also reinforced some patterns I’ve been using for a while now.

Pre-commit hooks as a contract #

I’ve never been too fond of pre-commit hooks. I’ve always been disciplined in using formatters and linters in projects, but never enjoyed having them set up to run automatically. This changed when I started working with agents.

I treat pre‑commit as a contract between me and the agent. The agent is instructed to always fix linting/testing failures before returning to me for input or declaring success. In particular, I’ve been using prek, which is a Rust rewrite of pre-commit.

Stricter lints for agents #

With agents, I’m also more inclined toward stricter, more pedantic lints to enforce properties like maximum line count in a function or file. When I’m writing code myself I make these changes instinctively. It’s obvious when a function becomes cumbersome when you’re editing by hand, but I noticed early on that an agent seems much more comfortable with 100+ line functions than I am.

If you care about such constraints (function length, etc.), encode them as lints rather than hoping the agent internalises your preferences.

As an aside, I’d never used flake-parts before, but its ability to automatically configure pre-commit hooks and formatting tools like treefmt in a Nix dev shell is really slick.

Self-updating instructions #

I hadn’t previously realised how good these tools are at writing their own instructions. I had been confused in the past at the length/complexity of AGENTS.md/CLAUDE.md files I’d seen, not realising that when you work with an agent to solve a problem, or remedy something it did, you can then prompt it to summarise what just happened in the CLAUDE.md file to prevent it from happening again.

Plan Mode #

To begin with I made extensive use of “Plan Mode”. The premise is that the agent can explore the codebase, read docs online and work with you to plan new features, but it cannot make changes to the code or your system.

This had been working well for me, but there were a couple of occasions where it seemed to get stuck in a “doom loop” trying to plan its way out of a complex failure.

On one particular occasion, this lasted for over an hour. I then switched to “Agent Mode”, gave it a follow-up prompt that instructed it to inspect a running application on Kubernetes directly using kubectl, and it was able to diagnose and solve the same problem in around 5 minutes. The problem here was that some “tools” are explicitly not permitted in “Plan Mode”, but depending on what you’re trying to achieve that can sometimes be a significant hindrance.

I’ve been favouring a different approach where I remain in “Agent Mode”, and prompt the agent that I’d like to work on a plan in a shared document (e.g. ./plans/new-feature.md), and that it may only edit that file until I say otherwise. I then work in a loop with the Agent where it writes a plan, I leave comments in-line in the file, and then it takes that into account - and repeat until I’m happy. Before asking the agent to implement the plan, I start a new session and ask it to implement the plan with fresh context. So far I’ve found this to be really effective.

Worktrees and Parallel Agents #

Pretty soon after starting work on more complicated features, I became interested in running parallel agents. Sometimes an agent can take a long time to work through a well-developed, and in that time I often felt I could be working my way through other items on my roadmap.

I initially tried to just run them concurrently on the same branch, but predictably they ended up fighting and making a bit of a mess. I’ve long been a fan of git worktrees, and they’re a perfect fit here - enabling me to run concurrent agents on separate instances of the codebase.

Even with worktrees, I still tend to work on orthogonal concerns so I don’t end up with big merge conflicts, but being able to work on a frontend feature, a backend feature and the CI pipeline all at once is a nice upgrade.

Visual prompting #

When working on frontend components, I built a nice workflow around screenshotting the UI, marking it up with Flameshot, then pasting the image back into Claude Code with the next prompt.

I found this to be a really effective way to drive iteration on web components. It felt closer to pair‑designing with a human: I’d circle spacing issues or misaligned elements, and the agent would propose CSS tweaks in response to what it saw.

Summary #

I had a blast building Brewlog, and I’ve been using it every day for about a month now. It’s different from how I’ve approached projects in the past, in a way that’s both liberating and terrifying in equal measures.

While some of the code in this project lacks the same attention to detail I might have applied, I likely would never have got around to building it without an agent.

I was surprised what the process taught me. Every day I hacked on Brewlog with Claude Code, I learned something. Whether that was something about Tailwind CSS, or an architectural pattern.

I think the key to useful output from these tools is that you remain the driver. You’re still responsible what you ship, which means giving the generated code the appropriate amount of review.

Brewlog is live at coffee.jnsgr.uk and runs comfortably on a free-tier instance on fly.io.

If you’re into speciality coffee and like owning your data, give it a try and let me know how you get on.

Finally, I’d like to thank the authors of Roastguide for the amazing work they’ve done on their app, the attention they pay to their community on Discord, and the fact that they were willing to get me a dump of my data I could use to populate the first deployment of Brewlog!

Supercharging Ubuntu Releases: Monthly Snapshots & Automation

Thu, 29 May 2025 00:00:00 +0000

This article was originally posted on the Ubuntu Discourse, and is reposted here. I welcome comments and further discussion in that thread.

Introduction #

Ubuntu has shipped on a predictable, six-month cadence for two decades. Twenty years ago, the idea of releasing an entire distribution every six months was considered forward looking, bold and even difficult. Things have changed since then: software engineering has evolved as a practice, and the advent of both rolling-release distributions like Arch Linux, and more recently image-based immutable distributions such as Universal Blue have meant that other projects with similar goals have adopted vastly different release models with some desirable properties.

My goal over the coming months is to build a release process that takes advantage of modern release engineering practices, while retaining the resilience and stability of our six-monthly releases. We’ll introduce significantly more automated testing, and ensure that the release process is transparent, repeatable and executable in a much shorter and well-known timeframe with little to no human intervention.

This journey will also create space for better system-wide testing, earlier detection of regressions, and a more productive collaboration with our community.

Monthly Snapshot Releases #

Starting in May 2025, we’re introducing monthly snapshot releases for Ubuntu.

Ubuntu is not “moving to monthly releases” or adopting a rolling release model; we’re committed to our six-monthly releases with a Long Term Support (LTS) release every two years. That doesn’t mean that our release process should be exempt from the same scrutiny that the rest of our engineering processes are subject to.

Today the Ubuntu Release process is the product of twenty years of evolution: it safeguards Ubuntu releases with a wealth of checks and balances, but is a largely manual process requiring significant human involvement.

The Ubuntu Release Team is a crowd of seasoned Ubuntu veterans who have been steadily releasing Ubuntu for many years. Many of this team are community members, some are or have been employed by Canonical in the past. More recently we have established the Canonical Ubuntu Release Management Team - a relatively new team at Canonical who’ll be collaborating with the Ubuntu Release Team to develop the new process.

To aid the Canonical team in their understanding of the existing processes, and the immovable requirements that sit beneath it, we’re introducing monthly snapshot releases for Ubuntu. These will not be fully-fledged releases of Ubuntu, but rather curated, testable milestones from our development stream. For the 25.10 (Questing Quokka) cycle, you can expect the following release schedule:

May 29, 2025: Questing Quokka - Snapshot 1
June 26, 2025: Questing Quokka - Snapshot 2
July 31, 2025: Questing Quokka - Snapshot 3
August 28, 2025: Questing Quokka - Snapshot 4
September 18, 2025: Questing Quokka - Beta
October 9, 2025: Questing Quokka - Final Release

This doesn’t mean you’ll start seeing Ubuntu versions off the six-month cadence. There will be no Ubuntu 25.07 or 25.08, etc. The monthly snapshots are exactly that: a snapshot of the development of Ubuntu 25.10. Snapshots are not meant for production use, but will help the release team move away from deep institutional knowledge, and toward clean well-documented automated workflows that are transparent, repeatable and testable.

With our current model, failure modes are not detected until they’re urgent and blocking an imminent release. The team conducts rigorous retrospectives on each release, but in my opinion it’s hard to meaningfully evolve such a process when it’s only exercised every six months. The monthly snapshots will create opportunities for us to test, understand and improve the process.

Embracing Automation #

One of the most valuable outcomes of this journey will be the opportunity to automate more of the process, freeing up time for the team to focus on more strategic tasks. Releasing a distribution is a complex process requiring coordination across architectures, images, mirrors, websites, testing infrastructure and even partner agreements. This also makes it hard to place a traditional CI tool at the heart of the process. As much as I like Github Actions, I think we’d quickly get lost trying to release Ubuntu with such a system, notwithstanding the fact that we’d lose control of the underlying infrastructure that releases Ubuntu.

I’ve been exploring the world of Durable Execution, which according to restate.dev is:

the practice of making code execution persistent, so that services recover automatically from crashes and restore the results of already completed operations and code blocks without re-executing them.

At Canonical, we’ve adopted Temporal in a few of our products and in many of our business processes. Temporal is a durable execution product that enables developers to solve complex distributed problems, but without being deep distributed systems experts. It’s a framework for composing tasks into workflows, with first-class primitives for dealing with failures, retries, exponential back-off and other concepts that enable the build of long-running complex workflows.

Having spent some time with Temporal myself, and watched other teams adopt it, I think it’s a great fit for engineering our next-generation release process. I want our engineers to focus on the logic of the release process, not the infrastructure behind it, and Temporal should enable them to do just that. The Temporal homepage sums it up nicely:

Write code as if failure doesn’t exist

Temporal workflows and activities can be written in many languages - and particularly in Python and Go. My expectation is that Go will prove to be an excellent fit for our process: it’s a fast and productive language that specialises in concurrency and asynchronous network operations, and has a powerful standard library containing much of the functionality we’ll need to build our new release process.

To take an overly simplistic view of how I expect this to go: we’ll take our existing release checklist, write a Go function for each step with some tests, and compose them together into one or more Temporal workflows that represent the full release process. This will take time, but this approach will enable us to incrementally demonstrate progress toward a fully-automated process over the coming cycles.

By making this move, not only will we make the process quicker, but also more observable, testable, reliable and easier to understand for everyone, not just the release team.

Improving Test Coverage #

One area I’d like to improve as a side-effect of this work is more full-system integration testing. Packages in the Ubuntu archive generally enjoy good coverage through a suite of autopkgtest tests, and there are numerous other places where integration tests are run on Ubuntu. With our traditional six-monthly cadence, full end-to-end testing of ISOs and the installer typically ramps up close to release time when changes are fewer (and riskier) and time is short.

With the introduction of monthly snapshots, we can integrate installer testing, full-disk encryption testing, graphical application testing and more as a regular, automated part of the release pipeline - not just as part of the development pipeline of each individual package. This means we should catch regressions earlier and surface more edge cases to be resolved before release.

One of the most important parts of increasing our testing culture is to make it clear where and how to contribute tests to Ubuntu. The easier we make it to write and contribute tests, the more tests we’re likely to add to the suite. We’re doing some work on this in parallel which will likely turn into a blog post of its own in the coming months.

In our current process, we have a heroic group of volunteers who kindly spend hours on our behalf testing the various flavours - exercising all the possible install paths and validating that what is about to be published is fit for purpose. I’d like to ensure that our volunteers’ time is spent as productively and rewardingly as possible, and I think we can automate much of this testing and allow them to focus on the more complex and nuanced aspects of each release, and raise the quality of Ubuntu across all the flavours.

What’s Next? #

We’re starting by modeling the current release process as it is. Once we’ve validated our assumptions about the current process, we’ll layer in improvements by reducing manual gates, parallelising independent steps, introducing more testing, and exercising the process each month to test (and measure) any improvements we’ve made.

My ultimate goal is a release system that’s incredibly “boring”: transparent, predictable, observable, and easy to reason about (even when things go wrong).

The new, fully-automated process will likely take several months to complete. When we think we’re done, we’ll do a release that runs both processes in parallel to ensure we get the outcome we expect before finally sunsetting the old process.

We’ll be building this work in the open (and hiring!) so if you’ve used Temporal in similar contexts, or are curious about contributing to this effort, we’d love to hear from you.

Until next time!

Adopting sudo-rs By Default in Ubuntu 25.10

Tue, 06 May 2025 00:00:00 +0000

This article was originally posted on the Ubuntu Discourse, and is reposted here. I welcome comments and further discussion in that thread.

Introduction #

Following on from Carefully But Purposefully Oxidising Ubuntu, Ubuntu will be the first major Linux distribution to adopt sudo-rs as the default implementation of sudo, in partnership with the Trifecta Tech Foundation

The change will be effective from the release of Ubuntu 25.10. You can see the Trifecta Tech Foundation’s announcement here.

What is `sudo-rs`? #

sudo-rs is a reimplementation of the traditional sudo tool, written in Rust. It’s being developed by the Trifecta Tech Foundation (TTF), a nonprofit focused on building secure, open source infrastructure components. The project is part of the Trifecta Tech Foundation’s Privilege Boundary initiative, which aims to handle privilege escalation with memory-safe alternatives.

The sudo command has long served as the defacto means of privilege escalation on Linux. As described in the original post, Rust provides strong guarantees against certain classes of memory-safety issues, which is pivotal for components at the privilege boundary.

The sudo-rs team is collaborating with Todd Miller, who’s maintained the original sudo for over thirty years. sudo-rs should not be considered a fork in the road, but rather a handshake across generations of secure systems. Throughout the development of sudo-rs, the TTF team have also made contributions to enhance the original sudo implementation.

The sudo-rs project is designed to be a drop in replacement for the original tool. For the vast majority of users, the upgrade should be completely transparent to their workflow. That said, sudo-rs is a not a “blind” reimplementation. The developers are taking a “less is more” approach. This means that some features of the original sudo may not be reimplemented if they serve only niche, or more recently considered “outdated” practices.

Erik Jonkers, Chair of the Trifecta Tech Foundation explains:

While no piece of software - in any language - is flawless, we believe the transition to Rust in systems programming is a vital step forward, it is very exciting to see Ubuntu committing to sudo-rs and taking the lead in moving the needle.

Sponsoring Mainstream Adoption #

Leading the mainstream adoption of a replacement to such a universally understood tool comes with responsibility. Before committing to ship sudo-rs in Ubuntu 26.04 LTS, we’ll test the transition in Ubuntu 25.10. We’re also sponsoring the development of some specific items, which has manifested as Milestone 5 in the upstream project:

Coarse-grained shell escape prevention (NOEXEC) on Linux (See PR #1073)
The ability to control AppArmor profiles (First PR #1067)
A sudoedit implementation
Support for Linux Kernels older than version 5.9

The final item may seem out of place, but because Ubuntu 20.04 LTS is still in support, without this work there could be situations where sudo fails to function if, for example, a 26.04 LTS OCI container was run on a 20.04 LTS host!

The team have also already begun work on ensuring that the test-suite is as compatible as possible with Ubuntu, to ensure any issues are caught early.

This isn’t just about shipping a new binary. It’s about setting a direction. We’re not abandoning C, or even rewriting all the utilities ourselves, but by choosing to replace one of the most security-critical tools in the system with a memory-safe alternative, we’re making a statement: resilience and sustainability are not optional in the future of open infrastructure.

Progress on `coreutils` #

Since the initial announcement, we’ve been working hard to more clearly define a plan for the migration to uutils coreutils in 25.10 and beyond. Similarly to our engagement with the Trifecta Tech Foundation, we’re also sponsoring the uutils project to ensure that some key gaps are closed before we ship 25.10. The sponsorship will primarily cover the development of SELinux support for common commands such as mv, ls, cp, etc.

The first step toward developing SELinux support was to add support for automated testing in Github Actions, since then the maintainers have begun work on the actual implementation.

The other feature we’re sponsoring is internationalisation support. At present, some of the utility implementations (such as sort) have an incomplete understanding of locales, and therefore may yield unexpected results. We expect that these two features should land in time for us to ship in 25.10, and we’ll continue to work with the uutils project throughout the 26.04 LTS cycle to close any remaining gaps we identify in the interim release.

One of the major concerns outlined in Julian’s post is about binary size. We’ve got a few tricks we can play here to get the size down, and there is already some conversation started upstream in Debian on how that might be achieved. There are also security implications, such as AppArmor’s lack of support for multi-call binaries. We’re currently working with the respective upstreams to discuss addressing this systematically, through in the interim we may need to build small wrapper binaries to enable compatibility with existing AppArmor profiles from the start.

Migration Mechanics #

Julian Klode posted recently on the Ubuntu Discourse outlining the packaging plan that will enable us both to migrate transparently to uutils coreutils, but also provide a convenient means for users to opt-out and switch back to GNU coreutils if they wish, or if they identify a gap in the new implementation. I expect this will be rare, but we want to make sure it’s as easy as possible to revert, and will be documenting this in detail before release.

Replacing coreutils isn’t as simple as swapping binaries. As an Essential package, its replacement must work immediately upon unpacking without relying on maintainer scripts, and without conflicting files across packages. To solve this, we’re introducing new coreutils-from-uutils and coreutils-from-gnu packages, as well as coreutils-from itself. For all the gory details, see the Discourse post!

The packaging work required to switch to sudo-rs is somewhat less complicated than with coreutils. The package is already available in Ubuntu (which you can still test on Ubuntu 24.04, 24.10 and 25.04 with oxidizr!), but unlike coreutils, sudo is not an Essential package, so we’ll be able to make use of the Debian alternatives system for the transition.

Summary #

Things are progressing nicely. We’ve established strong, productive relationships and are sponsoring work upstream to make these transitions viable.

We’ve got a strategy for migrating the default implementation of coreutils and sudo in Ubuntu 25.10 which will enable a seamless revert in cases where that is desired. While sudo-rs will be the default in 25.10, the original sudo will remain available for users who need it, and we’ll be gathering feedback to ensure a smooth transition before the 26.04 LTS.

Additionally, we’ve begun investigating the feasibility of providing SequoiaPGP and using it in APT instead of GnuPG. SequoiaPGP is a new OpenPGP library with a focus on safety and correctness, written in Rust. The GnuPG maintainers have recently forked the OpenPGP standard and are no longer compliant with it. Sequoia provides a modern alternative to GnuPG with strict behavior, and is already used in various other systems. More details to follow!

Stay tuned!

Revitalising Ubuntu Project Documentation

Tue, 01 Apr 2025 00:00:00 +0000

This article was originally posted on the Ubuntu Discourse, and is reposted here. I welcome comments and further discussion in that thread.

Introduction #

Back in February I wrote some thoughts on Ubuntu’s documentation and its role within the community. For a mostly-online software community, documentation is one of our most critical forms of communication.

In the last two years there has been lots of focus on the technical aspects of our documentation (how-to guides, tutorials, etc.), but I’d like to focus more on what I’m calling the “Ubuntu Project Documentation” over the coming months.

Documentation isn’t only about technical how-to guides and tutorials, nor is it only about troubleshooting or satisfying particular use-cases. Our documentation can set the tone for the project, give a means for the community to state an intent, and guide both current and future contributors in their daily work.

Ubuntu has a lot of documentation, most of which has grown organically over the last 20 years, but it’s not always easy to find or understand. Our documentation should illuminate and inspire a path to contribution. It should provide direction and clarity on complex issues, reference on technology and past decisions, and precision in the execution of process.

Our project documentation should detail what makes Ubuntu happen. How are decisions made? What are the teams contributing to Ubuntu? How are those teams appointed? What are their responsibilities? If you’re on the Main Inclusion Review (MIR) team and you’re assigned a package to review, what steps should you take? How does package sponsorship work, and who should you contact if you’re stuck? How are the Access Control Lists (ACLs) updated for packages and package sets, and who can make those changes? What does the journey look like from first time package bug-fixer to Ubuntu Core Developer?

These are all examples of questions that we, the collective conscious of Ubuntu, know the answers to, yet it is still difficult to find up-to-date answers to these questions, often requiring input from some of our busiest and most knowledgeable contributors to settle discussions and answer basic queries.

If a potential contributor identifies a bug in a package, there should be one authoritative source of information on where the package source can be located, how it can be pulled, built and tested, and how to work with a sponsor to land changes. Such a process is satisfying for contributors, making it more likely they’ll stay engaged, and therefore benefits the distribution’s longevity and sustainability.

Answering questions and mentoring people will remain a central part of our community’s role, but many of the first questions asked could be serviced by better documentation.

The Challenge #

Much of the required content already exists. The venerable Ubuntu Wiki was the go-to destination for such documentation, but has become outdated both technologically and in the content it serves. This degradation gained pace as we diversified the number of destinations that documentation could live: the Wiki, Discourse, Github, Launchpad, etc.

The Ubuntu Community team have made significant efforts over the past months to centralise the documentation for membership, our code of conduct and project governance. I also called out the renewed Stable Release Update (SRU) documentation in my first post for having made its first steps toward a new and improved structure.

These examples prove we have all the skills we need to write excellent documentation. Throughout the 25.10 cycle, I intend to put some focus on this, consolidating as much of our content into modern formats as possible, thereby making it as accessible as possible.

In doing this work, I hope to:

Illustrate contributor journeys across disciplines
Create resilience in the project by reducing the “bus factor”
Increase the accessibility and ergonomics of our documentation
Enable more efficient, asynchronous collaboration on a wide range of tasks

End Goal #

To quote the Canonical.com page on Documentation Practice:

we have embarked on a comprehensive, long-term project to transform documentation. Our aim is to create and maintain documentation product and practice that will represent a standard of excellence. We want documentation to be the best it possibly can be.

At the heart of this mission Diátaxis: a way of thinking about documentation. Diátaxis “prescribes approaches to content, architecture and form that emerge from a systematic approach to understanding the needs of documentation users”.

You’ll have seen Diátaxis in use across many of our product documentation pages: the Juju docs, the MAAS docs, the Pebble docs, the Rockcraft docs and many more.

Most of those existing sites are specific - they document a particular product or ecosystem which neatly scopes the documentation structure, but the Diátaxis framework can also be used to bring structure, precision and clarity to the documentation of the Ubuntu project as a whole.

Earlier this month I surveyed the various documentation sites in use by Canonical and the Ubuntu Community, and settled on three common themes around which we will structure our renewed project documentation:

Governance: in which membership, code of conduct, team structures, communication practices, delegation, mission, software licensing and 3rd-party software guidelines will be documented.
Develop Ubuntu: documentation for current and aspiring Ubuntu developers, including how to package software for Ubuntu, how to merge packages from Debian, how to sponsor packages, how to use git-ubuntu and conduct “+1 Maintenance”, etc.
Archive Administration: the nuts and bolts of managing Ubuntu’s prolific software repositories: how to manage seeds, configure phased updates, conduct an MIR, run an SRU process, etc.

These categories were not immediately obvious, and they’re not necessarily mutually exclusive, but they fell out quite naturally when trying to logically organise our existing content.

During the process, I came up with this rough sketch:

This illustrates how multiple categories of documentation from different corners of Ubuntu might come together in a single landing page. To give an idea of how we might further break down existing content by type, then category:

This may not be the final structure, but it’s indicative of how we can use Diátaxis to break down large documentation premises into smaller, more digestible and more ergonomic pieces.

The Plan #

During the Ubuntu 25.10 cycle, we’ll be dedicating two of our Technical Authors to make this happen. One of these authors has been largely responsible for overhauling the Ubuntu Server docs, but both are very familiar with Diátaxis and the tooling we using to deliver documentation.

Throughout this process, we’ll likely come across outdated, poorly reviewed or incorrect documentation, but as we work through the process of consolidating, we can note where this has happened and get it on our backlog to fix.

Perhaps we’ll find items which lend themselves to inclusion in the Canonical Open Documentation Academy, or maybe we’ll need to reach out to some of our less active community members for clarification, but once the structure is in place we’ll at least have a place to collaborate.

Once the transition is complete, there will be an authoritative source for project documentation that is easy to navigate, easy to contribute to and with a well-defined review process that encourages progress over gatekeeping.

Summary #

Documentation is the backbone of a thriving open-source community, guiding contributors, setting expectations, and ensuring long-term sustainability.

While Ubuntu has extensive documentation, much of it is scattered, outdated, or difficult to navigate. By leveraging the Diátaxis framework, we aim to bring structure, clarity, and accessibility to Ubuntu Project Documentation. Our focus will be on governance, development, and archive administration, ensuring that key processes and responsibilities are well-documented and easy to follow.

With dedicated technical authors and community collaboration, the Ubuntu 25.10 cycle will mark a significant step toward making our documentation searchable, structured, and sustainable.

I hope this effort will empower contributors, reduce reliance on institutional knowledge, and create a more resilient project for the next generation of Ubuntu developers and users.

Carefully But Purposefully Oxidising Ubuntu

Wed, 12 Mar 2025 00:00:00 +0000

This article was originally posted on the Ubuntu Discourse, and is reposted here. I welcome comments and further discussion in that thread.

Introduction #

Last month I published Engineering Ubuntu For The Next 20 Years, which outlines four key themes for how I intend to evolve Ubuntu in the coming years. In this post, I’ll focus on “Modernisation”. There are many areas we could look to modernise in Ubuntu: we could focus on the graphical shell experience, the virtualisation stack, core system utilities, default shell utilities, etc.

Over the years, projects like GNU Coreutils have been instrumental in shaping the Unix-like experience that Ubuntu and other Linux distributions ship to millions of users. According to the GNU website:

The GNU Core Utilities are the basic file, shell and text manipulation utilities of the GNU operating system. These are the core utilities which are expected to exist on every operating system.

This package provides utilities which have become synonymous with Linux to many - the likes of ls, cp, and mv. In recent years, there has been an effort to reimplement this suite of tools in Rust, with the goal of reaching 100% compatibility with the existing tools. Similar projects, like sudo-rs, aim to replace key security-critical utilities with more modern, memory-safe alternatives.

Starting with Ubuntu 25.10, my goal is to adopt some of these modern implementations as the default. My immediate goal is to make uutils’ coreutils implementation the default in Ubuntu 25.10, and subsequently in our next Long Term Support (LTS) release, Ubuntu 26.04 LTS, if the conditions are right.

But… why? #

Performance is a frequently cited rationale for “Rewrite it in Rust” projects. While performance is high on my list of priorities, it’s not the primary driver behind this change. These utilities are at the heart of the distribution - and it’s the enhanced resilience and safety that is more easily achieved with Rust ports that are most attractive to me.

The Rust language, its type system and its borrow checker (and its community!) work together to encourage developers to write safe, sound, resilient software. With added safety comes an increase in security guarantees, and with an increase in security comes an increase in overall resilience of the system - and where better to start than with the foundational tools that build the distribution?

I recently read an article about targeting foundational software with Rust in 2025. Among other things, the article asserts that “foundational software needs performance, reliability — and productivity”. If foundational software fails, so do all of the other layers built on top. If foundational packages have performance bottlenecks, they become a floor on the performance achievable by the layers above.

Ubuntu powers millions of devices around the world, from servers in your data centre, to safety critical systems in autonomous systems, so it behooves us to be absolutely certain we’re shipping the most resilient and trustworthy software we can.

There are lots of ways to achieve this: we can provide long term support for projects like Kubernetes, we can assure the code we write, and we can strive to achieve compliance with safety-centric standards, but another is by shipping software with the values of safety, soundness, correctness and resilience at their core.

That’s not to throw shade on the existing implementations, of course. Many of these tools have been stable for many years, quietly improving performance and fixing bugs. A lovely side benefit of working on newer implementations, is that it sometimes facilitates improvements in the original upstream projects, too!

I’ve written about my desire to increase the number of Ubuntu contributors, and I think projects like this will help. Rust may present a steeper learning curve than C in some ways, but by providing such a strong framework around the use of memory it also lowers the chances that a contributor accidentally commits potentially unsafe code.

Introducing `oxidizr` #

I did my homework before writing this post. I wanted to see how easy it was for me to live with these newer implementations and get a sense of their readiness for prime-time within the distribution. I also wanted a means of toggling between implementations so that I could easily switch back should I run into incompatibilities - and so oxidizr was born!

oxidizr is a command-line utility for managing system experiments that replace traditional Unix utilities with modern Rust-based alternatives on Ubuntu systems.

The oxidizr utility enables you to quickly swap in and out newer implementations of certain packages with relatively low risk. It has the notion of Experiments, where each experiment is a package that already exists in the archive that can be swapped in as an alternative to the default.

Version 1.0.0 supports the following experiments:

How does it work? #

Each experiment is subtly different since the paths of the utilities being replaced vary, but the process for enabling an experiment is generally:

Install the alternative package (e.g. apt install rust-coreutils)
For each binary shipped in the new package:
- Lookup the default path for that utility (e.g which date)
- Back up that file (e.g. cp /usr/bin/date /usr/bin/.date.oxidizr.bak)
- Symlink the new implementation in place (e.g. ln -s /usr/bin/coreutils /usr/bin/date)

There is also the facility to “disable” an experiment, which does the reverse of the sequence above:

For each binary shipped in the new package:
- Lookup the default path for the utility (e.g which date)
- Check for and restore any backed up versions (e.g cp /usr/bin/.date.oxidizr.bak /usr/bin/date)
Uninstall the package (e.g. apt remove rust-coreutils)

Thereby returning the system back to its original state! The tool is covered by a suite of integration tests which illustrate this behaviour which you can find on Github

Get started #

⚠️ WARNING ⚠️: oxidizr is an experimental tool to play with alternatives to foundational system utilities. It may cause a loss of data, or prevent your system from booting, so use with caution!

There are a couple of ways to get oxidizr on your system. If you already use cargo, you can do the following:

1

cargo install --git https://github.com/jnsgruk/oxidizr

Otherwise, you can download and install binary releases from Github:

1
2


# Download version 1.0.0 and extract to /usr/bin/oxidizr
curl -sL "https://github.com/jnsgruk/oxidizr/releases/download/v1.0.0/oxidizr_Linux_$(uname -m).tar.gz" | sudo tar -xvzf - -C /usr/bin oxidizr

Once installed you can invoke oxidizr to selectively enable/disable experiments. The default set of experiments in v1.0.0 is rust-coreutils and sudo-rs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Enable default experiments
sudo oxidizr enable
# Disable default experiments
sudo oxidizr disable
# Enable just coreutils
sudo oxidizr enable --experiments coreutils
# Enable all experiments without prompting with debug logging enabled
sudo oxidizr enable --all --yes -v
# Disable all experiments without prompting
sudo oxidizr disable --all --yes

The tool should work on all versions of Ubuntu after 24.04 LTS - though the diffutils experiment is only available from Ubuntu 24.10 onward.

The tool itself is stable and well covered with unit and integration tests, but nonetheless I’d urge you to start with a test virtual machine or a machine that isn’t your production workstation or server! I’ve been running the coreutils and sudo-rs experiments for around 2 weeks now on my Ubuntu 24.10 machines and haven’t had many issues (more on that below…).

How to Help #

If you’re interested in helping out on this mission, then I’d encourage you to play with the packages, either by installing them yourself or using oxidizr. Reply to the Discourse post with your experiences, file bugs and perhaps even dedicate some time to the relevant upstream projects to help with resolving bugs, implementing features or improving documentation, depending on your skill set.

You can also join us to discuss on our Matrix instance.

Next Steps #

Earlier this week, I met with @sylvestre to discuss my proposal to make uutils coreutils the default in Ubuntu 25.10. I was pleased to hear that he feels the project is ready for that level of exposure, so now we just need to work out the specifics. The Ubuntu Foundations team is already working up a plan for next cycle.

There will certainly be a few rough edges we’ll need to work out. In my testing, for example, the only incompatibility I’ve come across is that the update-initramfs script for Ubuntu uses cp -Z to preserve selinux labels when copying files. The cp, mv and ls commands from uutils don’t yet support the -Z flag, but I think we’ve worked out a way to unblock that work going forward, both in the upstream and in the next release of Ubuntu.

I’m going to do some more digging on sudo-rs over the coming weeks, with a view to assessing a similar transition.

Summary #

I’m really excited to see so much investment in the foundational utilities behind Linux. The uutils project seems to be picking up speed after their recent appearance at FOSDEM 2025, with efforts ongoing to rework procps, util-linux and more.

The sudo-rs project is now maintained by the Trifecta Tech Foundation, who are focused on “open infrastructure software in the public interest” . Their zlib-rs recently released v0.4.2, which appears to now be the fastest API-compatible zlib implementation. They’re also behind the Pendulum Project and ntpd-rs for memory-safe time synchronisation.

With Ubuntu, we’re in a position to drive awareness and adoption of these modern equivalents by making them either trivially available, or the default implementation for the world’s most deployed Linux distribution.

We will need to do so carefully, and be willing to scale back on the ambition where appropriate to avoid diluting the promise of stability and reliability that the Ubuntu LTS releases have become known for, but I’m confident that we can make progress on these topics over the coming months.

Engineering Ubuntu For The Next 20 Years

Tue, 11 Feb 2025 00:00:00 +0000

This article was originally posted on the Ubuntu Discourse, and is reposted here. I welcome comments and further discussion in that thread.

Introduction #

I’ve been a VP Engineering at Canonical for 3 years now, building Juju and our catalog of charms. In the last week of January, I was appointed the VP Engineering for Ubuntu at Canonical, where I will now oversee the Ubuntu Foundations, Server and Desktop teams.

Over the past 20 years, Ubuntu has become synonymous with “Linux” to many people. I fondly remember receiving my first Ubuntu CD in the post, shortly after my own Linux journey began in 2003 with booting Knoppix on a school computer. Throughout my career, Linux and open source have been prominent features that I’m very proud of. In the past few years I’ve made contributions to Ubuntu, Arch Linux, and more recently NixOS.

Ubuntu’s recent 20 year milestone is a timely reminder to pause and reflect on what made Ubuntu so exciting, so successful and so captivating to the Linux community. In 2004, the idea of releasing an operating system every six months was laughed off by many, but has now become the norm. Ubuntu builds upon Debian, aiming to bring the latest and very best open source had to offer to the masses. In the past 10 years, we’ve seen huge shifts in the way software is delivered - the success of large-scale cloud based operations necessitated a shift towards more automated testing, releasing and monitoring, and as the open source community around these projects grew, we had to evolve our ways of thinking, designing and communicating about software.

Four Key Themes #

As I step into this new role, I’ve reflected on how we can steer the engineering efforts behind Ubuntu. I’ve anchored this vision around four themes: Communication, Automation, Process and Modernisation.

Communication #

Communication is a central component of a distributed workforce - whether that workforce is employed by Canonical, members of our community or contributors from our partners. Ubuntu has relied for many years on mailing lists and IRC. These platforms enabled global teams to collaborate for years, and have been invaluable to the community. In 2025 we’re fortunate to have a wealth of communications platforms at our disposal, but we must use these tools strategically to avoid fragmentation.

On Jan 29 2025, the Ubuntu developer mailing list announced that the primary means of communication for Ubuntu developers will be the Ubuntu Community Matrix server. Matrix provides a rich, modern communications medium that is familiar to the next generation of engineers and tinkerers, who will be central to the continued progression of Ubuntu and open source. We’re in good company on Matrix, with many other Linux distributions and projects maintaining a presence on the platform. The recent migration of Ubuntu Forums to the Ubuntu Discourse, further consolidates the range of platforms we use to connect with one another.

To effect much of the change I’m describing in this post, we will need community support. I’ll be encouraging the leads of our internal teams in Ubuntu Foundations, Server and Desktop to be more forthcoming and regular with public updates that will serve two purposes: to share our intentions, progress and dreams for Ubuntu, but also to collaborate on refining our vision, ensuring we deliver a platform that is not just exciting, but relevant for many years to come.

Documentation is a critical form of communication. Our documentation enables our current users, but also illuminates the path for new contributors. Such documentation does exist, but much of it is fragmented across different platforms, duplicated and/or contradictory or simply difficult to find. As a company, and as a community, we must focus on ensuring both existing and potential contributors have access to the information they need on conventions, tools and processes. A good example of where this has already happened is the SRU documentation which was recently rebuilt in line with our documentation practices.

Automation #

Delivering a Linux distribution is a monumental task. With tens of thousands of packages across multiple architectures, the workload can be overwhelming - leaving little room for innovation until the foundational work is done. We’re fortunate to benefit from the diligent work done by the Debian community, yet there is a huge amount of work that goes into each Ubuntu release. One of our primary tasks as a distribution is package maintenance. While some may see this as menial or repetitive, it remains critical to the future of Ubuntu, and is a valuable specialist skill in its own right.

Software packaging is a complex and constantly evolving topic. Ubuntu relies heavily on a blend of Debian packages, and our own Snap packaging format. Debian packaging was revolutionary - responsible for huge advancements in the way we thought about delivering software, but as things have moved on some of those tools and practices are beginning to show their age.

I’d like to focus on enriching our build process with modern ideals and processes for automating the version bumps, testing, performance benchmarking and releasing of packages in the archive. High complexity tasks are error-prone and, without sufficient automation, risk becoming overly dependent on a few skilled individuals. We have the same challenge with Snaps, but they benefit from significantly more modern tooling as a consequence of the observations made about Debian packaging over many years.

The goal of this theme is not just to automate as much as possible (thereby increasing our collective capacity), but also to simplify processes where we can. Much of Ubuntu’s build process *is* automated, but those systems are disparate and often opaque to all but our most experienced contributors.

I’ve been inspired by how the NixOS community manages packaging. Every single package for the distro is represented as text files, in a single Git repository, with a universally observable continuous integration and integration testing pipeline (Hydra) that performs version bumps and simple maintenance tasks semi-autonomously. While this model carries its own challenges, there is something alluring about the transparency and accessibility of the systems that assemble, test and deliver software to their users.

Universal Blue, and by extension Project Bluefin, are recent additions to the Linux ecosystem that benefited from thinking hard about the tooling they use to build their distribution. They’ve centered their process around tools with which their cloud-native audience are already familiar.

My suggestion is not to imitate these projects, rather that the open source community is at its strongest when we collaborate and learn from one another. I think we can take inspiration from those surrounding us, and use that to inform our plans for Ubuntu’s future.

Process #

Process is closely tied to automation, but is frequently viewed negatively in software engineering, carrying connotations of bureaucracy and slowdowns. In my experience, a well-designed process empowers people to enact changes with confidence.

Ubuntu is built by all of us, in many countries and across all timezones. Concise, well-defined, lightweight processes promote autonomy and reduce uncertainty - enabling people to unblock themselves. Ubuntu is no stranger to process: the Main Inclusion Review (MIR), the aforementioned Stable Release Updates (SRU) process, the process for Snap store requests and many more have contributed to the success of Ubuntu, setting clear guardrails for contributors and ensuring we work to common standards.

My goal over the coming months is to work with you, the people behind Ubuntu, to identify which of these processes still serve us, and which need revising to simplify our work while maintaining our dedication to stability. I’ll consolidate the definitions of these processes, make them searchable, peer-reviewable, and more discoverable. Examples of where this has worked well are the Go proposal process, and the Ethereum Improvement Proposal process - both of which make it trivial to create, track and discuss proposals across the breadth of their respective projects.

If you submit an MIR, or work on an SRU, it should be trivial to understand the status of that request, and to communicate with the team executing that process where needed. If you’re interested in joining our community, it should be simple to get a sense of what is changing across the project, and where you might be able to help.

I’d like to tackle these problems and make these processes as transparent as possible.

Modernisation #

The world of computing has evolved dramatically in the last 20 years, and I’m proud that Ubuntu has continually adapted and thrived. In Linux alone there have been huge changes to what is considered “normal” for a Linux machine. Whether it be the introduction of `systemd`, the advent of languages with a focus on memory safety, the huge growth in virtualisation and containerisation technology, or even the introduction of Rust into the Linux kernel itself - the foundations of our distribution must be constantly assessed against the needs of our users.

I was proud to see the announcement last year that the Ubuntu Kernel team committed to shipping the very latest kernels in new versions of Ubuntu, wherever they possibly can. Even if that means shipping a kernel that’s in the release candidate phase, the team will stand by that kernel and continue to support it through the Ubuntu release’s life. While this could appear cavalier at a glance, what it represents is a willingness to rise to the challenge of shipping the very best of open source to our users. I’d like to see more of this. Ubuntu is a flagship Linux distribution and a starting point for many; we must ensure that our users are presented with the very best our community has to offer - even if that means a bit more hustle in the early days of a given release. This is of particular importance for our Long Term Support releases, which are relied upon by governments, financial institutions, educational establishments, nonprofits and many others for years after the initial release date.

We should look deeply at the tools we ship with Ubuntu by default - selecting for tools that have resilience, performance and maintainability at their core. There are countless examples in the open source community of tools being re-engineered, and re-imagined using tools and practices that have only relatively recently become available. Some of my personal favourites include command-line utilities such as eza, bat, and helix, the new ghostty terminal emulator, and more foundational projects such as the uutils rewrite of coreutils in Rust. Each of these projects are at varying levels of maturity, but have demonstrated a vision for a more modern Unix-like experience that emphasises resilience, performance and usability.

Another example of this is our work on TPM-backed full disk encryption, a project which promises encryption of our users’ data with no degradation to their user experience. This feature relies upon cryptographic hardware and techniques that have only recently become available to us, but enable us to deliver the potent combination of security and usability to our users.

Delivering Features #

What I’ve shared so far is a high-level overview, and many of the points under the four themes will take time to implement, with most appearing as a series of gradual improvements. You might be wondering whether we’ll focus on the latest trends and features, or prioritise that bug you reported.

While focusing on the latest trends or a single breakthrough feature can yield short-term progress, embracing these principles will create the space for sustained, impactful innovation.

That said, I’ve also been working on a list of incremental features and improvements that we can deliver in the coming months to enhance the Ubuntu experience. You’ll hear more from me and the team leads regularly as we share updates and progress.

Summary #

I’m incredibly excited to embark on this journey, and consider it a privilege to serve in this role. Together with the Ubuntu community, Canonical engineers, and our partners, we will build an open-source platform that enables the next 20 years of innovation in computing.

If you have ideas for the future of Ubuntu, or something in this post has resonated with you and you want to be involved either as a community member, or perhaps a future employee of Canonical, I’d love to hear from you.

Packaging the Multipass Flutter GUI for NixOS

Thu, 16 Jan 2025 00:00:00 +0000

Introduction #

The very first application I ever packaged in nixpkgs was Multipass. Multipass is, according to the website:

a CLI to launch and manage VMs on Windows, Mac and Linux that simulates a cloud environment with support for cloud-init.

I wrote in some detail about how I use Multipass and LXD on my workstation to create and manage VMs for testing and development. Last year Multipass shipped a new GUI written from the ground up in Flutter. It provides a clean, modern way to launch, manage and interact with VMs. When the GUI first shipped, I briefly attempted to get it to build with Nix, but some combination of my lack of knowledge, and the maturity of the Flutter tooling in nixpkgs at the time meant I never finished it.

As version 1.15.0 was in its release candidate stage, I decided to have another go!

This post will break down the process into steps, but if you’d like the tl;dr, take a look at the pull request.

Housekeeping #

Before getting started on packaging the GUI, I did some housekeeping on the Multipass package. First order of business was to simply bump the version to 1.15.0 and ensure the package still built, which it did.

Last year, RFC 140 was introduced to simplify the directory structure of nixpkgs, introducing a new pkgs/by-name directory which will (eventually) render pkgs/top-level/all-packages.nix useless. This new structure will make finding package definitions much easier, as they’ll all be arranged alphabetically by the first two characters of their name. Using the Multipass example, rather than pkgs/tools/virtualization/multipass/default.nix, the new preferred path would be pkgs/by-name/mu/multipass/package.nix. There was a talk at NixCon 2023 which summarised this nicely if you’d like more information.

As I was going to be carrying out some hefty work on Multipass by introducing the GUI, I took the opportunity to move to the new scheme as part of the changes.

Restructuring #

The Multipass package carries a few patches, and is already reasonably involved, so I decided that the best route forward was to create separate files for multipassd/multipass (the server & command line client binaries, written in C++, and already packaged) and multipass.gui (the new Flutter application). Thus the new structure for the package looks like so:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


.
# Derivation files
├── gui.nix
├── multipassd.nix
├── package.nix
# Lock file for Fluter app
├── pubspec.lock.json
# Update script for daemon/cli & GUI
├── update.sh
# Patches
├── cmake_no_fetch.patch
├── cmake_warning.patch
├── lxd_socket_path.patch
├── test_unreachable_call.patch
└── vcpkg_no_install.patch

Using `symlinkJoin` #

This package now takes a more unusual shape. Because the GUI is essentially a entirely separate application, written with a different toolkit in a different language, but I wanted the two to be installed as part of the overall multipass package, I decided to use two completely separate derivations and use symlinkJoin to bundle the two together. According to the manual, symlinkJoin:

can be used to put many derivations into the same directory structure. It works by creating a new derivation and adding symlinks to each of the paths listed. It expects two arguments, name, and paths. name (or alternatively pname and version) is the name used in the Nix store path for the created derivation. paths is a list of paths that will be symlinked. These paths can be to Nix store derivations or any other subdirectory contained within.

This is particularly useful for the Multipass package, because the Multipass GUI dynamically-links against libraries that are built as part of the multipassd derivation, and linking the two derivations in this way ensures that all those files are located correctly at runtime without any extra plumbing. This allowed me to keep the top-level package.nix relatively simple (annotated below):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55


{
# ... snip ...
}:
let
 name = "multipass";
 version = "1.15.0";

 # Fetch the source just once, to be used in both the multipassd and GUI derivations
 multipass_src = fetchFromGitHub {
 owner = "canonical";
 repo = "multipass";
 rev = "refs/tags/v${version}";
 hash = "sha256-RoOCh1winDs7BZwyduZziHj6EMe3sGMYonkA757UrIU=";
 fetchSubmodules = true;
 };

 # Shared metadata between derivations.
 commonMeta = {
 homepage = "https://multipass.run";
 license = lib.licenses.gpl3Plus;
 maintainers = with lib.maintainers; [ jnsgruk ];
 platforms = [ "x86_64-linux" ];
 };

 # Build the multipass server & client binaries from the shared source
 # and common metadata.
 multipassd = callPackage ./multipassd.nix {
 inherit commonMeta multipass_src version;
 };

 # Build the multipass GUI using shared source and common metadata.
 multipass-gui = callPackage ./gui.nix {
 inherit commonMeta multipass_src multipassd version;
 };
in
# Create the top-level `multipass` package using `symlinkJoin`
symlinkJoin {
 inherit version;
 pname = name;

 # Join both of the newly created derivations
 paths = [ multipassd multipass-gui ];

 # Ensure tests are run, and define a (new) update script.
 passthru = {
 tests = lib.optionalAttrs stdenv.hostPlatform.isLinux {
 inherit (nixosTests) multipass;
 };
 updateScript = ./update.sh;
 };

 meta = commonMeta // {
 description = "Ubuntu VMs on demand for any workstation";
 };
}

Building the GUI #

Normally the server, client and GUI binaries are all built as part of a single invocation of cmake. We can’t do that here, so the early code I put into the multipass package when I failed to package the GUI before still stood - essentially stopping cmake from trying to build the Flutter GUI:

1
2
3
4


{
 # We'll build the flutter application separately using buildFlutterApplication
 cmakeFlags = [ "-DMULTIPASS_ENABLE_FLUTTER_GUI=false" ];
}

Next, I created gui.nix which uses the pkgs.buildFlutterApplication helper. The derivation is annotated below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67


{
 # Passed in from the package.nix shown above
 commonMeta,
 multipass_src,
 multipassd,
 version,
 # ... snip ...
}:
flutter327.buildFlutterApplication {
 inherit version;
 pname = "multipass-gui";
 src = multipass_src;

 # Ensure we build in the correct directory for the GUI code.
 sourceRoot = "${multipass_src.name}/src/client/gui";

 # To make the builds as deterministic as possible, nixpkgs reads the pubspec.lock
 # file for the Flutter application and downloads/caches the correct versions of
 # flutter packages to be used in the build. It does this with `pub2nix`.
 #
 # Here I use a JSON representation of the pubspec.lock file committed into nixpkgs.
 pubspecLock = lib.importJSON ./pubspec.lock.json;

 # Some of the Flutter dependencies used are pulled from Github (branches) directly, so this
 # ensures that the versions/commits are pinned for future builds.
 gitHashes = {
 dartssh2 = "sha256-2pypKwurziwGLZYuGaxlS2lzN3UvJp3bRTvvYYxEqRI=";
 hotkey_manager_linux = "sha256-aO0h94YZvgV/ggVupNw8GjyZsnXrq3qTHRDtuhNv3oI=";
 system_info2 = "sha256-fly7E2vG+bQ/+QGzXk+DYba73RZccltdW2LpZGDKX60=";
 tray_menu = "sha256-riiAiBEms+9ARog8i+MR1fto1Yqx+gwbBWyNbNq6VTM=";
 window_size = "sha256-71PqQzf+qY23hTJvcm0Oye8tng3Asr42E2vfF1nBmVA=";
 xterm = "sha256-h8vIonTPUVnNqZPk/A4ZV7EYCMyM0rrErL9ZOMe4ZBE=";
 };

 # ... snip ...

 # The Multipass GUI relies on protobuf for API communication; ensure the
 # Dart protobuf definitions are compiled before we build.
 preBuild = ''
 mkdir -p lib/generated

 # Generate the Dart gRPC code for the Multipass GUI.
 protoc \
 --plugin=protoc-gen-dart=${lib.getExe protoc-gen-dart} \
 --dart_out=grpc:lib/generated \
 -I ../../rpc \
 ../../rpc/multipass.proto \
 google/protobuf/timestamp.proto
 '';

 # Libraries built by multipassd are linked dynamically at runtime; multipassd must
 # be installed for the GUI to function.
 runtimeDependencies = [ multipassd ];

 # Housekeeping to ensure the icons/desktop files are correctly installed.
 postFixup = ''
 mv $out/bin/multipass_gui $out/bin/multipass.gui

 install -Dm444 $out/app/multipass-gui/data/flutter_assets/assets/icon.png \
 $out/share/icons/hicolor/256x256/apps/multipass.gui.png

 cp $out/share/applications/multipass.gui.autostart.desktop \
 $out/share/applications/multipass.gui.desktop
 '';

 # ... snip ..
}

Update Script #

That was enough to ensure that the GUI worked correctly. My concern now lied with the maintenance of the package. The process for updating the derivation now involved:

Bumping the version string
Updating the source hashes
Updating the pubspec.lock.json file
Updating the gitHashes of the Flutter dependencies

This seemed a little convoluted and error prone to handle manually each time, so I decided to implement an update script for the package to take care of future version bumps. Thankfully nixpkgs has good provision for this, allowing package maintainers to specify passthru.updateScript to a derivation, which I did here.

I won’t post the full update script here, as it’s lots of unsightly bash! Essentially the script takes care of determining the latest version released on Github by using curl, then updates the pubspec.lock.json using a combination of curl/yj, then finally works through updating the source hash for the Multipass repo, gRPC fork it uses, and each of the git hashes for the Flutter dependencies.

This can be invoked from within the nixpkgs repo with:

1

nix-shell maintainers/scripts/update.nix --argstr package multipass

Summary #

Overall, this turned out cleaner than I expected, though it took me longer to figure out than this post might suggest. My hope is that this will a useful reference to anybody trying to package Flutter applications with Nix, and if you’re a user of the multipass package, check out the shiny new GUI that’s available to you! The GUI is available from 1.15.0 onwards, which at the time of writing means you’ll need to install the package from nixos-unstable or similar (sadly I didn’t land the change in time for NixOS 24.11).

In my opinion, the Multipass team have done a lovely job on the GUI - and for many users this will provide a much improved onboarding experience. Nice work 😎

Experimenting with Rust, Nix, K6 and Parca

Sun, 01 Dec 2024 00:00:00 +0000

Introduction #

Over the past couple of weeks, I’ve been teaching myself Rust. I don’t have a pressing need to write much Rust right now, but I’m intrigued by the promises of memory safety, and have been increasingly impressed at the quality of some of the software that the community produces. I also think that the concepts popularised by Rust, such as the borrow checker, will stick around in computing for many years to come and I’d like to have more hands-on experience with that.

The Rust language also encourages the ideas of safety and soundness - sound code is (approximately) code that can’t cause memory corruption or exhibit undefined behaviour. You can read more in this excellent post Safety and Soundness in Rust.

This blog post started out life as a post about Rust and my experience learning it, but I got interested in the performance of the server implementation I came up with and the post evolved into a post more about profiling and load testing!

Learning Rust #

It’s been a little while since I dug into a new programming language, so I thought I’d mention how I went about it. In general I’m someone who learns best “by doing”. I normally try to read through some of the basic concepts, then jump to a project that will enable me to exercise them quite quickly. Fortunately, Rust has an excellent official guide in the form of the Rust Book, which covers everything from the obligatory Hello, World!, to concurrency, memory safety, publishing packages on crates.io and more. I thoroughly recommend it.

I’ve also picked up a copy of Rust for Rustaceans which was recommended by a couple of different colleagues - I intend to work through this next.

So, after working through the Rust Book over the course of about a week in my spare time, I needed a project!

Rewriting `gosherve` in Rust #

In my first post on this blog I talked about a small Go project I wrote several years ago named gosherve. This was one of my first Go projects - a simple web server that can serve some static assets, and a set of short-links/redirects which are specified in a Github Gist. It’s been happily running my website for several years, and it felt like a small, but ambitious enough project for my first adventure into Rust - particuarly as over the years gosherve has grown Prometheus metrics and the ability to serve assets from an embedded filesystem.

As I anticipated, naming the new project was the hardest part. I landed on servy, at least for now. I’m reasonably happy with the code - at the time of writing:

It can serve redirects
It can serve embedded web assets
It provides a metrics server on a separate port
It has reasonable unit/integration test coverage

One of the things I love about Go is the built-in ability to embed static assets into a binary through the //go:embed directive, which gives you a pointer to an embedded filesystem. I was able to achieve a similar effect in Rust with axum-embed, which in turn builds upon rust-embed.

I used Nix to create different variants of the build (i.e. a “vanilla” build, and one that serves my website by creating a derivation just for the web content, and another for the jnsgruk binary).

You can run a copy of this website (frozen in time just before I wrote this post!) by running nix run github:jnsgruk/servy#jnsgruk, or build a container for it by running nix build github:jnsgruk/servy#jnsgruk-container.

I was expecting the Rust binary to be quite a lot smaller. I’m not sure why. The old (Go) binary weighs in at 68MB, where the new Rust binary comes in at 67MB.

So I was right! I guess?! The result here is relatively uninteresting - a good chunk of both binaries is just the static assets (images!) that make up this site. At the time of writing the jnsgruk-content derivation evaluates at around 57MB - meaning there is 10MB and 9MB respectively for Go and Rust added by the actual server code.

Basic Load Testing with `k6` #

I’ve been looking for an excuse to play with k6 for a while. According to the documentation:

Grafana k6 is an open-source, developer-friendly, and extensible load testing tool. k6 allows you to prevent performance issues and proactively improve reliability.

In very simple terms, to use k6 you define a script (in Javascript) that outlines a set of requests to make, their success criteria and (optionally) the strategy for ramping up load on the server. It has many more features than I used for this project, but I was impressed with how simple it was to get started. I began by running the following (grabbing k6 from nixpkgs):

1
2
3
4
5


# Start a shell with k6 available
❯ nix shell github:NixOS/nixpkgs/nixos-unstable#k6
# Init a new k6 script
❯ k6 new
Initialized a new k6 test script in script.js. You can now execute it by running `k6 run script.js`.

Now I just needed to work out what I wanted to test! I wanted a relatively simple test that requested a mix of web assets and redirects for a sustained period, to see how much throughput I could achieve with each of the server implementations. The template came with some sensible setup for the number of VUs (virtual users) and duration of the test:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


import { check } from "k6"
import http from "k6/http"

export const options = {
 // A number specifying the number of VUs to run concurrently.
 vus: 10,
 // A string specifying the total duration of the test run.
 duration: "30s",
}
// ...

VUs are an emulation of a user interacting with your service; each of them is an agent which will execute the test script. Each time the script is executed (by a VU), that’s known as an “iteration”.

I wanted to test a variety of requests, so I first generated a list of valid URLs for my server. I used cariddi which is a web crawler written in Go, combined with jq to create a list of valid paths in a JSON file:

1
2
3
4
5
6
7


# Start a shell with k6 available
❯ nix shell github:NixOS/nixpkgs/nixos-unstable#cariddi
# Run cariddi and generate the paths.json
❯ echo http://localhost:8080 \
 | cariddi -- -plain \
 | cut -d"/" -f4- \
 | jq -r -nR '[inputs | select(length>0)]' > paths.json

Similarly I generated a list of valid redirects:

1
2
3


❯ curl -s "https://gist.githubusercontent.com/jnsgruk/b590f114af1b041eeeab3e7f6e9851b7/raw" \
 | cut -d" " -f1 \
 | jq -r -nR '[inputs | select(length>0)]' > redirects.json

Now in my k6 script, I was able to read those files to create a list of URLs to batch GET during the load test:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


// Allow the host to be overridden by the 'K6_HOST' env var.
const host = __ENV.K6_HOST || "http://localhost:8080"
// Read the list of paths/redirects.
const assets = JSON.parse(open(`./paths.json`))
const redirects = JSON.parse(open(`./redirects.json`))

export default function () {
 // Batch requests to redirects and static assets.
 const responses = http.batch(
 [...assets, ...redirects].map((p) => {
 return ["GET", `${host}/${p}`, null, { redirects: 0 }]
 })
 )

 // For each response, ensure we get a 200/301/308/404
 // response - we shouldn't see anything else.
 responses.forEach((r) => {
 check(r, {
 status: (res) => {
 return [200, 301, 308, 404].includes(res.status)
 },
 })
 })
}

I’m using a pretty naive check function here, though they can be expanded to check for other conditions in the response body, the size of the response, etc.

You can see my final test script on Github.

Finally, I ran k6 and let it rip! This initial run was against servy, running on my workstation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40


❯ k6 run script.js

 /\  Grafana /‾‾/
 /\  / \  |\  __ / /
 / \/ \  | |/ / / ‾‾\
 / \  | ( | (‾) |
 / __________ \  |_|\_\  \_____/

 execution: local
 script: script.js
 output: -

 scenarios: (100.00%) 1 scenario, 10 max VUs, 1m0s max duration (incl. graceful stop):
 * default: 10 looping VUs for 30s (gracefulStop: 30s)

INFO[0030] [k6-reporter v2.3.0] Generating HTML summary report source=console

 ✓ status

 checks.........................: 100.00% ✓ 1331889 ✗ 0
 data_received..................: 148 GB 4.9 GB/s
 data_sent......................: 163 MB 5.4 MB/s
 http_req_blocked...............: avg=2.41µs min=390ns med=1.72µs max=17.3ms p(90)=3.6µs p(95)=4.91µs
 http_req_connecting............: avg=4ns min=0s med=0s max=462.21µs p(90)=0s p(95)=0s
 http_req_duration..............: avg=501.25µs min=42.08µs med=139.77µs max=45.41ms p(90)=291.59µs p(95)=419.65µs
 { expected_response:true }...: avg=501.25µs min=42.08µs med=139.77µs max=45.41ms p(90)=291.59µs p(95)=419.65µs
 http_req_failed................: 0.00% ✓ 0 ✗ 1331889
 http_req_receiving.............: avg=379.39µs min=4.13µs med=29.07µs max=45.23ms p(90)=108.31µs p(95)=177.05µs
 http_req_sending...............: avg=4.45µs min=960ns med=3.48µs max=3.36ms p(90)=6.83µs p(95)=8.79µs
 http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
 http_req_waiting...............: avg=117.4µs min=33.65µs med=98.19µs max=23.03ms p(90)=176.34µs p(95)=225.09µs
 http_reqs......................: 1331889 44333.219428/s
 iteration_duration.............: avg=38.08ms min=4.92ms med=45.46ms max=80.35ms p(90)=47.68ms p(95)=48.53ms
 iterations.....................: 7881 262.326742/s
 vus............................: 10 min=10 max=10
 vus_max........................: 10 min=10 max=10


running (0m30.0s), 00/10 VUs, 7881 complete and 0 interrupted iterations
default ✓ [======================================] 10 VUs 30s

Nice! Between the 10 VUs we configured, k6 managed nearly 8000 iterations, and my new server responded with a total of 148GB of data!

Nix-ified Load Testing #

Now I’d figured out the basics of k6, I wanted to create some infrastructure that would allow me to run load tests in consistent environments against both old and new implementations of my server.

A couple of ideas came to mind here - the first of which was the NixOS integration test driver. I absolutely love this feature of Nix, and the scripts for interacting with the driver are nice and simple to maintain. One slight irritation in this case is that the test machines don’t have access to the internet - which is where the redirects map is fetched from in my precompiled server binaries. It’s certainly possible to get clever with a fake redirects server and some DNS shennanigans in the test machines, but I opted instead to build simple nixosConfigurations which could be started as VMs, similar to the approach taken in a previous post.

I wanted to automate:

The creation of a VM with both the Go and Rust versions of my server
The generation of the paths.json and redirects.json files I created above
Running the k6 load test
Fetching the results from the load test

Defining Test VMs #

I created a new repository with a flake.nix which took nixpkgs, my jnsgr.uk repo and my servy repo as inputs, pinned to the latest revisions at the time of writing:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


{
 description = "web server benchmarking flake";

 inputs = {
 nixpkgs.url = "github:nixos/nixpkgs/nixpkgs-unstable";
 jnsgruk-go.url = "github:jnsgruk/jnsgr.uk/3240a0104c01ae672a6f5f7b0529ad08bcbc8af2";
 jnsgruk-rust.url = "github:jnsgruk/servy/71a337317defc779c6c55d486e20d104c5d478f2";
 };

 # ...
}

Next up I needed a virtual machine definition, which I first defined as a nixosConfiguration in the flake’s outputs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


{
 # ... inputs ...
 outputs = { self, nixpkgs, jnsgruk-go, jnsgruk-rust, ... }:
 let
 forAllSystems = nixpkgs.lib.genAttrs [ "x86_64-linux" "aarch64-linux" ];
 in
 {
 # A minimal NixOS virtual machine which used for testing craft applications.
 nixosConfigurations = forAllSystems (system: {
 benchvm = nixpkgs.lib.nixosSystem {
 specialArgs = {
 inherit self system;
 jnsgruk-go = jnsgruk-go.packages.${system}.jnsgruk;
 jnsgruk-rust = jnsgruk-rust.packages.${system}.jnsgruk;
 };
 modules = [ ./vm.nix ];
 };
 });
 };
}

The actual machine configuration lives in vm.nix. This is a standard NixOS configuration, with the addition of some elements that define the specs of the virtual machine used to boot it (cores, memory, disk, etc.). In this case the machine is configured with just a root user, and the password password. This is not an ideal setup from a security standpoint, but these VMs were only run on my workstation (behind NAT) for a short period of time and the plain text password eased the automation I’m about to describe. A more robust approach would have been to put my SSH public keys into the authorized keys definition for the user, but then you folks wouldn’t have been able to play along as easily!

In this particular configuration, the VM is set up like so:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


virtualisation = {
 forwardPorts = [{
 from = "host";
 host.port = 2222;
 guest.port = 22;
 }];
 cores = 4;
 memorySize = 4096;
 diskSize = 10000;
};

With that in place, we can now boot the VM (in the background):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


❯ nix run .#nixosConfigurations.benchvm.config.system.build.vm -- --daemonize --display none
❯ ssh -p 2222 root@localhost
The authenticity of host '[localhost]:2222 ([127.0.0.1]:2222)' can't be established.
ED25519 key fingerprint is SHA256:icnH3EQAzmjdfCkyPWFljQWaVSCaXdP2M+ekKXd0NlY.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[localhost]:2222' (ED25519) to the list of known hosts.
(root@localhost) Password:

[root@benchvm:~]#

Wrapping `k6` #

Now on to automating the test itself. When configuring the VM, I made sure to include a systemd unit for each of the server implementations, so next I wanted to write a simple script that would execute the load test for each implementation and fetch the results.

Before that, I modified the k6 script to not only output a report to stdout, but also to a text file, a JSON file and a rendered HTML file (I didn’t know at the time which I’d prefer, and I wanted to survey the options!). I achieved this by defining the handleSummary function in the k6 script:

1
2
3
4
5
6
7
8


export function handleSummary(data) {
 return {
 "summary.json": JSON.stringify(data),
 "summary.html": htmlReport(data),
 "summary.txt": textSummary(data, { enableColors: false, indent: "" }),
 stdout: "\n" + textSummary(data, { enableColors: true, indent: "" }) + "\n\n",
 }
}

First I scripted the generation of the paths and the execution of the test script, and packaged the combination up as a Nix package for installation into the test machine. This is just a bash script which ensures only the specified implementation is running (with systemd), generates the list of URLs to test and then runs k6 against the server. That script is packaged as a Nix package called k6test which contains both my bash script and the k6 test script; the Nix package is then added to the VM configuration.

Automating Test Execution #

I borrowed the vm-exec script from a past project, renaming it benchvm-exec, and created a new script called benchvm-test:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


#!/usr/bin/env bash
set -euo pipefail
info() { echo -e "\e[92m[$HOSTNAME] $*\e[0m"; }

# Set some SSH options to ignore host key errors and make logging quieter.
# This is a bad idea in general, but here is used to faciliate comms with
# a brand new VM each time.
SSH_OPTS=(-P 2222 -o "UserKnownHostsFile=/dev/null" -o "StrictHostKeyChecking=no")

info "Running k6test against $1"
benchvm-exec k6test "$1"

info "Collecting results files"
sshpass -ppassword scp "${SSH_OPTS[@]}" root@localhost:summary.json "summary-${1}.json"
sshpass -ppassword scp "${SSH_OPTS[@]}" root@localhost:summary.html "summary-${1}.html"
sshpass -ppassword scp "${SSH_OPTS[@]}" root@localhost:summary.txt "summary-${1}.txt"

info "Results available in summary-${1}.txt, summary-${1}.json and summary-${1}.html"

This was the final piece of the puzzle for now; after building and packaging this script and defining the test VM as a package in the flake, I’d enabled the following workflow:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Start the devShell for the server-bench project
❯ nix develop github:jnsgruk/server-bench
# Build & run the VM
❯ run-benchvm-vm --daemonize --display none
# Start, and load test the gosherve based server
❯ benchvm-test jnsgruk-go
# Start, and load test the Rust based server
❯ benchvm-test jnsgruk-rust
# All done, power down the VM
❯ benchvm-exec poweroff

Excellent! Now on to some actual testing!

Initial Load Test Results #

Having seen the results for servy earlier, I started up the existing gosherve based server and ran the same test, only to discover quite a delta in the results. Where servy managed 4.9 GB/s throughout the test, gosherve only achieved 697 MB/s (sending 21GB in total). The full results are in the details box below, but overall it managed much lower performance across the board on the measurements that k6 makes.

Initial gosherve load-test results

checks.........................: 100.00% ✓ 189449 ✗ 0
data_received..................: 21 GB 697 MB/s
data_sent......................: 26 MB 848 kB/s
http_req_blocked...............: avg=2.02µs min=400ns med=1.42µs max=19.76ms p(90)=2.29µs p(95)=2.77µs
http_req_connecting............: avg=203ns min=0s med=0s max=19.52ms p(90)=0s p(95)=0s
http_req_duration..............: avg=9.3ms min=29.49µs med=9.36ms max=152.39ms p(90)=13.16ms p(95)=14.76ms
{ expected_response:true }...: avg=9.3ms min=29.49µs med=9.36ms max=152.39ms p(90)=13.16ms p(95)=14.76ms
http_req_failed................: 0.00% ✓ 0 ✗ 189449
http_req_receiving.............: avg=106.56µs min=4.88µs med=48.7µs max=6.01ms p(90)=215.06µs p(95)=381.38µs
http_req_sending...............: avg=4.81µs min=950ns med=3.97µs max=19.71ms p(90)=5.93µs p(95)=6.98µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=9.19ms min=22.85µs med=9.26ms max=152.2ms p(90)=12.96ms p(95)=14.51ms
http_reqs......................: 189449 6281.256789/s
iteration_duration.............: avg=268.14ms min=176.88ms med=250.61ms max=433.23ms p(90)=334.14ms p(95)=342.1ms
iterations.....................: 1121 37.1672/s
vus............................: 10 min=10 max=10
vus_max........................: 10 min=10 max=10

I was quite surprised by this: my expectation was that the Go HTTP server would outperform what I’d put together in Rust. I decided to look a little deeper and see if I could figure out why, or at least why the delta was so big.

Profiling `gosherve` with Parca #

For last couple of years, I’ve been playing with Parca, which is a continuous profiling tool written in Go by the folks at Polar Signals. I wrote about it back in 2022 on Charmhub, demonstrating how Parca could be used for profiling applications deployed with Juju.

Instrumenting `gosherve` #

There are two components to the open source Parca offering - the server backend, and a zero-instrumentation, eBPF-based agent which can be used for on-CPU profiling of any workload. Because gosherve is Go based, I didn’t need the agent so long as I enabled the pprof endpoint in my server, which is trivial for almost any Go application. I took main.go and applied the following changes to ensure that I could hit the pprof endpoints on port 6060:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


diff --git a/main.go b/main.go
index 964ceb2..374c440 100644
--- a/main.go
+++ b/main.go
@@ -10,6 +10,9 @@ import (
 "log/slog"
 "os"

+ "net/http"
+ _ "net/http/pprof"
+
 "github.com/jnsgruk/gosherve/pkg/logging"
 "github.com/jnsgruk/gosherve/pkg/server"
 )
@@ -29,6 +32,11 @@ func main() {
 flag.Parse()
 logging.SetupLogger(*logLevel)

+ go func() {
+ http.ListenAndServe(":6060", nil)
+ slog.Info("pprof server started on port 6060")
+ }()
+
 // Create an fs.FS from the embedded filesystem
 fsys, err := fs.Sub(publicFS, "public")
 if err != nil {

Next I fired up Parca (having first packaged it for NixOS 😉) and configured it to scrape the new jnsgruk binary while I ran the same test against it, which resulted in the following profile (which you can explore yourself on pprof.me):

Looking at this, we can see that around a third of the time was spent in garbage collection. This seemed high; it was more prominent on this run than others but it was persistently a high portion of the CPU time. Something to loop back to! However, inside the route handler itself there are two things that stood out to me:

Initial optimisations #

In the red box, you can see about a third of the time spent in the route handler was in creating SHA1 sums, which is happening here to calculate the ETag for content before serving it. This could probably be optimised - either by changing how the ETag is calculated (i.e. not hashing the whole file contents), or by caching ETags in memory, particularly given that in the case of my website, gosherve only serves assets which are embedded (read only) in the binary, so the ETag will never change for a given file for a particular compiled binary.

In the blue box, we can see that another third of the time spent is on refreshing the redirects. This seemed strange; the redirects don’t change throughout the life of the test, and we’re only requesting known redirects from our redirects.json file. This seemed much more likely to be a culprit in the poor load test results, because unlike calculating a SHA1 sum (which is almost entirely CPU-bound), this function is making an outbound (TLS) connection to Github and potentially blocking on IO before parsing the response, rechecking the redirects map for a match, etc.

As I looked at the code, I noticed an obvious (and somewhat embarrassing!) mistake I’d made when implementing gosherve. Because the server tries to parse redirects before looking for files with a matching path, every request for a file causes the redirects map to be updated from its upstream source 🤦. Looking back at the new servy implementation, I hadn’t made the same mistake - perhaps this was the reason the Rust version was so much faster?

I made a small change to gosherve and re-ran the test, which resulted in this profile and quite a substantial bump in request performance!

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


checks.........................: 100.00% ✓ 913783 ✗ 0
data_received..................: 101 GB 3.4 GB/s
data_sent......................: 123 MB 4.1 MB/s
http_req_blocked...............: avg=3.21µs min=390ns med=2.12µs max=14.12ms p(90)=4.19µs p(95)=5.7µs
http_req_connecting............: avg=70ns min=0s med=0s max=14.09ms p(90)=0s p(95)=0s
http_req_duration..............: avg=1.8ms min=42.12µs med=899.13µs max=31.31ms p(90)=4.86ms p(95)=6.45ms
 { expected_response:true }...: avg=1.8ms min=42.12µs med=899.13µs max=31.31ms p(90)=4.86ms p(95)=6.45ms
http_req_failed................: 0.00% ✓ 0 ✗ 913783
http_req_receiving.............: avg=261.88µs min=4.85µs med=63.59µs max=17.8ms p(90)=620.15µs p(95)=1.31ms
http_req_sending...............: avg=8.21µs min=1.05µs med=5.21µs max=8.73ms p(90)=9.87µs p(95)=13.07µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=1.53ms min=30.81µs med=656.71µs max=30.76ms p(90)=4.38ms p(95)=5.91ms
http_reqs......................: 913783 30435.133157/s
iteration_duration.............: avg=55.5ms min=28.86ms med=55.18ms max=77.07ms p(90)=61.02ms p(95)=62.98ms
iterations.....................: 5407 180.089545/s
vus............................: 10 min=10 max=10
vus_max........................: 10 min=10 max=10

The overall throughput increased by five times, with a decrease in latency across the board. Still around a third less overall throughput than the Rust server, but a huge improvement nonetheless.

I subsequently experimented with (naively) removing the ETag calculation to see whether or not it was worth implementing some caching - but it actually resulted in very little difference in throughput and CPU utilisation. Besides, while the calculation takes some CPU time, in a real-life deployment it reduces the overall load on the server by ensuring that http.ServeContent only sends actual content when there is a new version, relying more heavily on the user’s browser cache. My k6 tests (deliberately) weren’t sending the right headers (such as If-None-Match) to benefit from the behaviour one might expect from a browser, and it was just requesting the same files over and over again with the equivalent of a cold/empty cache.

Reducing Allocations #

Now back to all those allocations, and the time spent in garbage collection as a result… gosherve was spending about a third of it’s time in the io.ReadFile function, and given the amount of time spent in garbage collection, any optimisation made to the number of allocations on the heap would likely yield another big performance increase.

The problem lied in the routeHandler, where I was reading the entire contents of each file that was being served into memory. fs.ReadFile makes an allocation the size of the file it’s reading, meaning the contents of the file end up on the heap. In a situation like our load test - this means the entire contents of my website ended up on the heap, and the Go runtime was busily garbage collecting to clean up.

I looked around at alternative ways to implement the same functionality in a more efficient manner. In Go 1.22, http.ServeFileFS was introduced as a counterpart to http.ServeFile. http.ServeFileFS operates on an fs.FS rather than the host filesystem. Under the hood, rather than reading the whole file into memory ServeFileFS is using io.Copy and thus io.copyBuffer which allocates a single buffer that fits on the stack - fewer allocations means less memory usage, and less time spent occupying cores with garbage collection!

The change to gosherve was pretty simple - you can see a slightly shortened diff below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


diff --git a/pkg/server/route_handler.go b/pkg/server/route_handler.go
index 2417e45..e0c9d68 100644
--- a/pkg/server/route_handler.go
+++ b/pkg/server/route_handler.go
@@ -1,7 +1,6 @@
 // snip!

- // Try reading the file and return early if that fails
- b, err := fs.ReadFile(*s.webroot, filepath)
- if err != nil {
- return false
- }
-
 w.Header().Set("Cache-Control", "public, max-age=31536000, must-revalidate")
- w.Header().Set("ETag", fmt.Sprintf(`"%d-%x"`, len(b), sha1.Sum(b)))
+ w.Header().Set("ETag", fmt.Sprintf(`"%s-%d-%x"`, fi.Name(), fi.Size(), sha1.Sum([]byte(fi.ModTime().String()))))

- http.ServeContent(w, r, filepath, time.Now(), bytes.NewReader(b))
+ http.ServeFileFS(w, r, *s.webroot, filepath)
 s.metrics.responseStatus.WithLabelValues(strconv.Itoa(http.StatusOK)).Inc()
 l.Info("served file", slog.Group("response", "status_code", http.StatusOK, "file", filepath))

This change also necessitated a change to ETag calculation, which now only hashes the filename, size and modified time. There is a slight oddity in that files from an embedded filesystem have their timestamps all fixed (to avoid changes in static files messing with build reproducibility), but since the files cannot change during a binary’s lifetime, the ETag calculated from the modified timestamp is still just as valid.

I quickly made a new build of jnsgruk using a locally referenced copy of gosherve and ran k6/parca to check the outcome. You can explore the profile yourself on pprof.me:

Here you can clearly see that the runtime is spending much less time in garbage collection, and the call to io.Copy/io.copyBuffer without the calls to fs.ReadFile. It’s now also much harder to spot the ETag generation - likely because we’re now only hashing a small string, rather than the whole file contents.

The k6 results came back as follows 🚀:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


checks.........................: 100.00% ✓ 1525300 ✗ 0
data_received..................: 170 GB 5.7 GB/s
data_sent......................: 207 MB 6.9 MB/s
http_req_blocked...............: avg=4µs min=410ns med=2.36µs max=18.47ms p(90)=5.02µs p(95)=6.77µs
http_req_connecting............: avg=40ns min=0s med=0s max=17.98ms p(90)=0s p(95)=0s
http_req_duration..............: avg=999.6µs min=42.17µs med=645.03µs max=24.22ms p(90)=2.34ms p(95)=3.09ms
 { expected_response:true }...: avg=999.6µs min=42.17µs med=645.03µs max=24.22ms p(90)=2.34ms p(95)=3.09ms
http_req_failed................: 0.00% ✓ 0 ✗ 1525300
http_req_receiving.............: avg=245.43µs min=4.64µs med=75.29µs max=23.48ms p(90)=626.08µs p(95)=1.18ms
http_req_sending...............: avg=9.69µs min=1µs med=5.49µs max=19.54ms p(90)=10.86µs p(95)=14.46µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=744.47µs min=28.09µs med=395.81µs max=20.62ms p(90)=1.89ms p(95)=2.58ms
http_reqs......................: 1525300 50823.211595/s
iteration_duration.............: avg=34.41ms min=11.91ms med=34.44ms max=67.75ms p(90)=37.6ms p(95)=38.61ms
iterations.....................: 8716 290.418352/s
vus............................: 10 min=10 max=10
vus_max........................: 10 min=10 max=10

The Go server is now in the lead! There isn’t a huge margin here between the Rust server and the Go server, but this is about the sort of difference I was expecting based on the reading I’d done before doing the testing.

Profiling `servy`? #

I did profile servy using parca-agent, but I’ve yet to look at the results in detail. The profile is a lot more complex (see below), partially as a result of how much work is going on under the hood when standing up an axum server with hyper, tower and tokio. I’m going to spend some more time with this profile over the coming weeks, but you can explore it for yourself on pprof.me.

If you do play with the profile on pprof.me, try filtering by function name and entering servy, and you’ll see the parts of the code I wrote highlighted on the profile to explore:

By the time you read this, the parca-agent should be upstream into nixpkgs, but in the mean time you can pull the package I included in the server-bench repo and use that:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Start the devShell for the server-bench project
❯ nix develop github:jnsgruk/server-bench
# Start the agent
❯ sudo parca-agent -- \
 --remote-store-address="grpc.polarsignals.com:443" \
 --remote-store-bearer-token="$TOKEN" \
 --node="$HOSTNAME"

# Or, on Ubuntu (see polarsignals.com/docs/setup-collection-snaps)
❯ sudo snap install --classic parca-agent

In my Nix example above, I’m sending profiles to Polar Signals Cloud, but you can also set the remote-store-address to the address of a locally running parca instance no problem.

Depending on the language you’re profiling, you’ll either need to include debug info in the binary you’re profiling (which is what I did), or upload your source/debug info manually. There are more instructions for that in the docs.

Final Results & Comparisons #

I wanted to gather some test results from all three targets across a number of different machine configurations; my website runs on a teeny-tiny Fly.io instance with 1 vCPU and 256MB RAM, so I was interested to see how performance changed on machines with varying configurations. I’ve included a summary in the table below which highlights just the data_received measurement from each test, and you can see the full reports from k6 by expanding the details box:

Expand for full `k6` reports

1 vCPU/256MB RAM/1 VUs - gosherve (old)

checks.........................: 100.00% ✓ 23322 ✗ 0
data_received..................: 2.6 GB 86 MB/s
data_sent......................: 3.1 MB 105 kB/s
http_req_blocked...............: avg=2.41µs min=400ns med=1.28µs max=1.04ms p(90)=1.87µs p(95)=2.27µs
http_req_connecting............: avg=50ns min=0s med=0s max=508.13µs p(90)=0s p(95)=0s
http_req_duration..............: avg=7.57ms min=49.54µs med=7.71ms max=143.13ms p(90)=9.42ms p(95)=10.65ms
{ expected_response:true }...: avg=7.57ms min=49.54µs med=7.71ms max=143.13ms p(90)=9.42ms p(95)=10.65ms
http_req_failed................: 0.00% ✓ 0 ✗ 23322
http_req_receiving.............: avg=98.03µs min=4.46µs med=57.25µs max=7.69ms p(90)=177.19µs p(95)=250.6µs
http_req_sending...............: avg=5.7µs min=970ns med=3.1µs max=480.07µs p(90)=4.79µs p(95)=9.26µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=7.47ms min=37.26µs med=7.62ms max=142.9ms p(90)=9.24ms p(95)=10.33ms
http_reqs......................: 23322 776.881114/s
iteration_duration.............: avg=217.52ms min=187.06ms med=198.66ms max=359.52ms p(90)=276.89ms p(95)=280.19ms
iterations.....................: 138 4.59693/s
vus............................: 1 min=1 max=1
vus_max........................: 1 min=1 max=1

1 vCPU/256MB RAM/1 VUs - gosherve

checks.........................: 100.00% ✓ 158691 ✗ 0
data_received..................: 18 GB 587 MB/s
data_sent......................: 21 MB 714 kB/s
http_req_blocked...............: avg=1.89µs min=379ns med=1.08µs max=1.15ms p(90)=1.62µs p(95)=1.98µs
http_req_connecting............: avg=11ns min=0s med=0s max=655.05µs p(90)=0s p(95)=0s
http_req_duration..............: avg=1.04ms min=48.25µs med=537.57µs max=31.33ms p(90)=2.63ms p(95)=3.66ms
{ expected_response:true }...: avg=1.04ms min=48.25µs med=537.57µs max=31.33ms p(90)=2.63ms p(95)=3.66ms
http_req_failed................: 0.00% ✓ 0 ✗ 158691
http_req_receiving.............: avg=91.12µs min=3.66µs med=20.8µs max=23.05ms p(90)=156.29µs p(95)=287.17µs
http_req_sending...............: avg=6.47µs min=900ns med=2.57µs max=2.22ms p(90)=4.06µs p(95)=6.21µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=949.02µs min=35.88µs med=466.14µs max=31.3ms p(90)=2.46ms p(95)=3.47ms
http_reqs......................: 158691 5287.057691/s
iteration_duration.............: avg=31.95ms min=21.16ms med=30.39ms max=107.46ms p(90)=36.19ms p(95)=41.18ms
iterations.....................: 939 31.284365/s
vus............................: 1 min=1 max=1
vus_max........................: 1 min=1 max=1

1 vCPU/256MB RAM/1 VUs - servy

checks.........................: 100.00% ✓ 148213 ✗ 0
data_received..................: 16 GB 547 MB/s
data_sent......................: 18 MB 605 kB/s
http_req_blocked...............: avg=1.5µs min=350ns med=971ns max=1.11ms p(90)=1.48µs p(95)=1.82µs
http_req_connecting............: avg=11ns min=0s med=0s max=451.97µs p(90)=0s p(95)=0s
http_req_duration..............: avg=928.79µs min=69.84µs med=671.38µs max=55.59ms p(90)=1.57ms p(95)=2.18ms
{ expected_response:true }...: avg=928.79µs min=69.84µs med=671.38µs max=55.59ms p(90)=1.57ms p(95)=2.18ms
http_req_failed................: 0.00% ✓ 0 ✗ 148213
http_req_receiving.............: avg=136.98µs min=3.76µs med=17.14µs max=46.31ms p(90)=103.7µs p(95)=192.52µs
http_req_sending...............: avg=5.33µs min=890ns med=2.35µs max=2.99ms p(90)=3.69µs p(95)=5.05µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=786.47µs min=55.89µs med=629.47µs max=17.92ms p(90)=1.43ms p(95)=1.96ms
http_reqs......................: 148213 4940.080272/s
iteration_duration.............: avg=34.19ms min=16.57ms med=25.68ms max=109.45ms p(90)=55.08ms p(95)=57.83ms
iterations.....................: 877 29.231244/s
vus............................: 1 min=1 max=1
vus_max........................: 1 min=1 max=1

2 vCPU/4GB RAM/10 VUs - gosherve (old)

checks.........................: 100.00% ✓ 116272 ✗ 0
data_received..................: 13 GB 426 MB/s
data_sent......................: 16 MB 518 kB/s
http_req_blocked...............: avg=2.25µs min=380ns med=1.06µs max=4.3ms p(90)=1.69µs p(95)=2.05µs
http_req_connecting............: avg=475ns min=0s med=0s max=4.27ms p(90)=0s p(95)=0s
http_req_duration..............: avg=15.37ms min=29.8µs med=14.28ms max=145.82ms p(90)=24.79ms p(95)=29.78ms
{ expected_response:true }...: avg=15.37ms min=29.8µs med=14.28ms max=145.82ms p(90)=24.79ms p(95)=29.78ms
http_req_failed................: 0.00% ✓ 0 ✗ 116272
http_req_receiving.............: avg=174.41µs min=4.15µs med=28.84µs max=21.95ms p(90)=241.72µs p(95)=703.97µs
http_req_sending...............: avg=7.22µs min=890ns med=2.98µs max=5.58ms p(90)=5.8µs p(95)=9.15µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=15.19ms min=22.98µs med=14.11ms max=144.5ms p(90)=24.48ms p(95)=29.37ms
http_reqs......................: 116272 3837.442876/s
iteration_duration.............: avg=439.41ms min=323.27ms med=430.04ms max=567.74ms p(90)=505.87ms p(95)=516.96ms
iterations.....................: 688 22.706763/s
vus............................: 10 min=10 max=10
vus_max........................: 10 min=10 max=10

2 vCPU/4GB RAM/10 VUs - gosherve

checks.........................: 100.00% ✓ 362167 ✗ 0
data_received..................: 40 GB 1.3 GB/s
data_sent......................: 49 MB 1.6 MB/s
http_req_blocked...............: avg=2.23µs min=360ns med=770ns max=12.31ms p(90)=1.34µs p(95)=1.66µs
http_req_connecting............: avg=117ns min=0s med=0s max=1.84ms p(90)=0s p(95)=0s
http_req_duration..............: avg=4.7ms min=32.38µs med=3.94ms max=47.81ms p(90)=9.04ms p(95)=11.63ms
{ expected_response:true }...: avg=4.7ms min=32.38µs med=3.94ms max=47.81ms p(90)=9.04ms p(95)=11.63ms
http_req_failed................: 0.00% ✓ 0 ✗ 362167
http_req_receiving.............: avg=196.28µs min=3.38µs med=15.42µs max=25.62ms p(90)=142.41µs p(95)=896.21µs
http_req_sending...............: avg=8.44µs min=890ns med=2.19µs max=18.13ms p(90)=3.63µs p(95)=6.34µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=4.5ms min=25.57µs med=3.77ms max=45.46ms p(90)=8.74ms p(95)=11.13ms
http_reqs......................: 362167 12040.737924/s
iteration_duration.............: avg=140.21ms min=80.54ms med=139.82ms max=221.91ms p(90)=153.25ms p(95)=157.95ms
iterations.....................: 2143 71.24697/s
vus............................: 10 min=10 max=10
vus_max........................: 10 min=10 max=10

2 vCPU/4GB RAM/10 VUs - servy

checks.........................: 100.00% ✓ 464412 ✗ 0
data_received..................: 51 GB 1.7 GB/s
data_sent......................: 57 MB 1.9 MB/s
http_req_blocked...............: avg=1.99µs min=360ns med=720ns max=14.38ms p(90)=1.13µs p(95)=1.42µs
http_req_connecting............: avg=92ns min=0s med=0s max=6.71ms p(90)=0s p(95)=0s
http_req_duration..............: avg=3.67ms min=44.91µs med=3.07ms max=60.17ms p(90)=5.9ms p(95)=7.98ms
{ expected_response:true }...: avg=3.67ms min=44.91µs med=3.07ms max=60.17ms p(90)=5.9ms p(95)=7.98ms
http_req_failed................: 0.00% ✓ 0 ✗ 464412
http_req_receiving.............: avg=206.58µs min=3.41µs med=14.06µs max=56.96ms p(90)=56.97µs p(95)=143.43µs
http_req_sending...............: avg=7.12µs min=870ns med=1.62µs max=14.84ms p(90)=3.01µs p(95)=5.78µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=3.45ms min=35.61µs med=3.01ms max=24.64ms p(90)=5.7ms p(95)=7.54ms
http_reqs......................: 464412 15448.582291/s
iteration_duration.............: avg=109.3ms min=63.59ms med=108.88ms max=161.77ms p(90)=120.71ms p(95)=124.65ms
iterations.....................: 2748 91.41173/s
vus............................: 10 min=10 max=10
vus_max........................: 10 min=10 max=10

16 vCPU/16GB RAM/10 VUs - gosherve (old)

checks.........................: 100.00% ✓ 119990 ✗ 0
data_received..................: 13 GB 439 MB/s
data_sent......................: 16 MB 534 kB/s
http_req_blocked...............: avg=2.05µs min=440ns med=1.55µs max=972.62µs p(90)=2.32µs p(95)=2.74µs
http_req_connecting............: avg=38ns min=0s med=0s max=275.54µs p(90)=0s p(95)=0s
http_req_duration..............: avg=14.82ms min=35.49µs med=15.16ms max=148.26ms p(90)=21.12ms p(95)=23.89ms
{ expected_response:true }...: avg=14.82ms min=35.49µs med=15.16ms max=148.26ms p(90)=21.12ms p(95)=23.89ms
http_req_failed................: 0.00% ✓ 0 ✗ 119990
http_req_receiving.............: avg=104.7µs min=5.07µs med=43.71µs max=8.96ms p(90)=228.22µs p(95)=396.05µs
http_req_sending...............: avg=4.52µs min=940ns med=3.99µs max=944.81µs p(90)=6.28µs p(95)=7.86µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=14.71ms min=23.97µs med=15.05ms max=148.25ms p(90)=20.95ms p(95)=23.7ms
http_reqs......................: 119990 3957.203279/s
iteration_duration.............: avg=426.57ms min=343.33ms med=422.38ms max=526.97ms p(90)=491ms p(95)=502.47ms
iterations.....................: 710 23.415404/s
vus............................: 10 min=10 max=10
vus_max........................: 10 min=10 max=10

16 vCPU/16GB RAM/10 VUs - gosherve

checks.........................: 100.00% ✓ 1297244 ✗ 0
data_received..................: 144 GB 4.8 GB/s
data_sent......................: 175 MB 5.8 MB/s
http_req_blocked...............: avg=2.83µs min=419ns med=1.63µs max=7.97ms p(90)=2.95µs p(95)=3.78µs
http_req_connecting............: avg=2ns min=0s med=0s max=142.69µs p(90)=0s p(95)=0s
http_req_duration..............: avg=1.22ms min=31.96µs med=697.97µs max=36.9ms p(90)=3.16ms p(95)=4.07ms
{ expected_response:true }...: avg=1.22ms min=31.96µs med=697.97µs max=36.9ms p(90)=3.16ms p(95)=4.07ms
http_req_failed................: 0.00% ✓ 0 ✗ 1297244
http_req_receiving.............: avg=234.98µs min=4.21µs med=39.77µs max=35.6ms p(90)=640.71µs p(95)=1.24ms
http_req_sending...............: avg=7.19µs min=930ns med=3.91µs max=30.11ms p(90)=6.84µs p(95)=9.22µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=980.01µs min=19.74µs med=465.75µs max=16.64ms p(90)=2.75ms p(95)=3.68ms
http_reqs......................: 1297244 43221.02723/s
iteration_duration.............: avg=39.08ms min=23.12ms med=38.84ms max=76ms p(90)=43.87ms p(95)=45.71ms
iterations.....................: 7676 255.745723/s
vus............................: 10 min=10 max=10
vus_max........................: 10 min=10 max=10

16 vCPU/16GB RAM/10 VUs - servy

checks.........................: 100.00% ✓ 1407770 ✗ 0
data_received..................: 156 GB 5.2 GB/s
data_sent......................: 172 MB 5.7 MB/s
http_req_blocked...............: avg=2.28µs min=420ns med=1.64µs max=3.96ms p(90)=3µs p(95)=3.94µs
http_req_connecting............: avg=2ns min=0s med=0s max=292.26µs p(90)=0s p(95)=0s
http_req_duration..............: avg=512.52µs min=40.21µs med=177.39µs max=45.65ms p(90)=444.41µs p(95)=623.84µs
{ expected_response:true }...: avg=512.52µs min=40.21µs med=177.39µs max=45.65ms p(90)=444.41µs p(95)=623.84µs
http_req_failed................: 0.00% ✓ 0 ✗ 1407770
http_req_receiving.............: avg=324.43µs min=4.02µs med=26.56µs max=45.31ms p(90)=88.04µs p(95)=147.94µs
http_req_sending...............: avg=4.33µs min=969ns med=3.35µs max=4.11ms p(90)=5.86µs p(95)=7.31µs
http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_waiting...............: avg=183.74µs min=31.45µs med=132.09µs max=8.96ms p(90)=344.35µs p(95)=465.04µs
http_reqs......................: 1407770 46854.747614/s
iteration_duration.............: avg=36.03ms min=5.19ms med=45.31ms max=77.29ms p(90)=48.06ms p(95)=48.95ms
iterations.....................: 8330 277.247027/s
vus............................: 10 min=10 max=10
vus_max........................: 10 min=10 max=10

	`gosherve (old)`	`gosherve`	`servy`
1 vCPU/256MB RAM/1 VUs	86MB/s	587MB/s	547MB/s
2 vCPU/4GB RAM/10 VUs	426MB/s	1.3GB/s	1.7GB/s
16 vCPU/16GB RAM/10 VUs	439MB/s	4.8GB/s	5.2GB/s

Interestingly, it seems that the servy variant actually outperformed gosherve again in these VM-based tests, where it didn’t when run directly on my machine. In any case, the gap is now much smaller between the two, and gosherve still seems to maintain an edge on very small machines.

Note I had to drop the VUs to 1 on the smallest iteration, or I was seeing all three of the variants getting OOM-killed!

I also added a new flake input in the server-bench repo that adds the gosherve based repo again, but pinned to the pre-optimised version.

As such, if you want to try reproducing any of these results and measure old vs. new vs. servy, you just need to do the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Start the devShell for the server-bench project
❯ nix develop github:jnsgruk/server-bench
# Build & run the VM
❯ run-benchvm-vm --daemonize --display none
# Edit the core/memory count as required
❯ vim vm.nix
# Start, and load test a server, choosing your variety
❯ benchvm-test <jnsgruk-go-old|jnsgruk-go|jnsgruk-rust>
# Power down the VM
❯ benchvm-exec poweroff

The results will be gathered in a collection of summary-*.<txt|html|json> files for you to inspect.

Summary #

This was supposed to be a blog about learning Rust, but I’m pleased with where it ended up! This was by no means a particularly in-depth dive into measuring the performance of web servers, focusing almost entirely on the request throughput. There are other factors that could be taken into consideration - such as the resulting memory pressure on the machine, CPU usage, etc. I may come back to this in another post with more comprehensive measurement of both.

Hopefully this post illustrated the power of load testing and profiling tools - even when only used at quite a surface level, and demonstrated how Nix can be used to create robust structures around existing tools to help with such workflows.

I owe a thank you to Frederic from Polar Signals, who gave me a bunch of helpful tips along the way while I was using Parca and profiling gosherve. After my initial improvement in the request handler, Frederic helped me track down and remove the use of fs.ReadFile and further improve the performance of gosherve.

I’m still very early in my Rust development, so if you’ve spotted something horrible in servy, or you’ve got experience with k6 and parca, then reach out and let me know! I’d love to see the creative ways people use these tools.

The performance gap between servy and gosherve is now quite narrow. Close enough, in fact, that they seem to win over one another depending on different conditions in the test environment, and never by a particularly high margin. I haven’t migrated this site over yet, but perhaps I will in the near future. Besides, Fly.io are already kindly limiting the damage that can be done by too many requests for me!

See you next time!

Hot Tub Monitoring with Home Assistant and ESPHome

Tue, 12 Nov 2024 00:00:00 +0000

Introduction #

This year we put a wood-fired hot tub in our back garden. It’s a total indulgence and a real treat. I originally purchased the tub for my wife, but I must admit to having grown quite fond of it myself!

The tub itself is made from Canadian Redwood Cedar. The photo below was taken shortly after it was assembled, so there is still a little bit of leakage which stopped a few days later once the wood had expanded. The 30kW wood burner is the most effective way to heat the tub (which holds around 1700l of water), but there is also a small electric heater in line with the pump & filter which are stashed behind the tub.

There is something wonderfully simple and low-tech about the whole arrangement, but naturally I wanted to get an understanding of the energy usage, and get a reading on the temperature so that I could more accurately set it up for when we wanted to use it. Part of the reasoning for understanding the energy usage was to ensure that the pump and UV filter only fire up when the house is generating enough solar energy to cover it.

A picture paints a thousand words, so you can see what I’m talking about below:

Energy Usage Monitoring #

Starting with the easy part: monitoring the energy usage for each of the components. What you can’t see in the picture above is the following:

An Elecro Vulcan 3kw heater
An evoUV 15w UV clarifier
A Crystal Enterprises CC2513 pump
A Crystal Enterprises CC3030 filter unit

The UV filter, heater and sand filter are all essentially “in line” - when the pump is switched on, water is pulled from the bottom of the tub, through the UV clarifier, sand filter and heater, and then back into the tub (not necessarily in that order!). The heater has a thermostat in it so that it can cut in/out depending on the desired temperature as the water passes through.

In reality, the 3kw heater would take 12-15 hours to heat the tub from scratch, which is why we use the burner that usually takes around 1.5-2 hours. The electric heater is good for maintaining the temperature once the initial heating has been done.

I’ve had quite a lot of success with TP-Link Tapo P110 smart plugs. They’re very cheap, and there is already a competent Home Assistant integration for them. The only downside is that they require a proprietary app for initial setup. After having bought a few of these, I learned about Tasmota, and will probably buy smart plugs compatible with this or similar open source firmwares in the future.

My friends from the Self Hosted Podcast seem to recommend buying pre-flashed smart devices from cloudfree.shop, which I’ll certainly consider next time.

All of the above said - the TP-Link plugs work great for this use-case. I’ve got one for the pump and one for the heater. The pump is configured to run for 2 hours a day during peak daylight hours, and the heater can be toggled as and when I need it. The result, when added to Home Assistant is some neat energy monitoring and the ability to toggle things on and off easily when I need:

In an alternative view that summarises power usage for my home, there is a neat section covering the usage from individual devices (all connected to smart plugs):

As you can see - the heater is incredibly power-hungry, and as a result we hardly use it. In the two hours it ran on the day shown above, it consumed almost twice the energy that my main workstation used all day. We now constrain it’s use to a couple of hours per day in the summer months when solar generation is at its peak - otherwise we just heat the tub with the wood burner as we want.

Temperature Sensor: Initial Solution #

When the tub originally arrived, I looked around for an internet-connected pool thermometer, and was quite surprised at the lack of options. I had assumed this was something relatively common. I ended up ordering an Inkbird IBS-P02R kit.

The kit comprises a small floating thermometer, and a display unit that communicates wirelessly with the thermometer. I picked this particular model because the display unit can be connected to Wi-Fi and then checked on through an app. I had (naively…) assumed that I’d be able to get access through some sort of API, but that turned out not to be the case.

Overall, this solution worked okay. The temperature is reported in 5/10/15 minute intervals according to the configuration, and I had no issues with range/connectivity even though the tub is some distance from my house (and the house is heavily insulated with triple glazing!).

The Inkbird app for iOS leaves a lot to be desired, though. I found the user experience pretty frustrating. The web service was often slow to respond, and it required an annoying number of clicks to get to the information I wanted.

While researching the different ways I could get access to the information, I re-discovered the Inkbird IBS-P01B which is a very similar thermometer to the one I’d bought, but communicates over bluetooth. I’d initially ruled this out since my server is well out of bluetooth range, but I’d also been looking for an excuse to play with an ESP32-based microcontroller…

Temperature Sensor: Homemade Solution #

After a little bit of research I ended up ordering the following:

1x Inkbird IBS-P01B thermometer
1x ESP32-WROOM-32U development board

The former was ordered from AliExpress, and the latter from Amazon. I chose the ESP32-WROOM-32U specifically because it has an external antenna. Given the eventual placement of the ESP32 and the location of the hot tub, I wanted to maximise the chances of establishing a good bluetooth connection.

This was to be my first foray into ESP32/Arduino type development. I’d read (and heard…) lots about ESPHome, and given my goal was to integrate with Home Assistant, this felt like the right route.

Connecting the ESP32 #

The first step was to flash ESPHome onto the device. The development board I bought comes with a CP2012 USB to UART bridge on the board, so connecting the device to my machine was as simple as plugging it in with a MicroUSB cable.

Before the device can be flashed, it needs to be put into programming mode. In my case, the board has a handy BOOT button. The process for entering programming mode was therefore (starting with the device unplugged and powered down):

Press and hold the BOOT button
Connect the ESP32 over USB
Wait a few seconds
Release the BOOT button

Other development boards may not have a button, but all it really does is bridge GPIO0 and GND on the board which can be done with a wire, too. There are good docs on the connection process on the ESPHome website.

ESPHome Support #

ESPHome doesn’t natively support the IBS-P01B, but after some reading it seemed that the underlying messaging format/protocol is similar enough to the IBS-TH1 and IBS-TH2 that the same configuration can be used.

I also came across this blog post where the author had manually created an ESPHome configuration that appeared to work, and with the exact same hardware.

Nonetheless - to my mind, simpler is better most of the time, so I set about creating a simple configuration using the supported ESPHome platform.

ESPHome Dashboard #

There are a few different ways to get started with ESPHome. You can “install” using Docker, which seems to be the preferred method (passing through /dev/ttyUSBx to the container for USB connectivity), or manually using Python packages.

I tried to follow the instructions to get started with Home Assistant, but it seems this is not compatible with “unsupervised” Home Assistant servers like mine.

The esphome package is readily available in nixpkgs, so I was able to get started and fire up the dashboard like so:

❯ nix run unstable#esphome -- dashboard .
2024-11-12 14:47:19,072 INFO Starting dashboard web server on http://0.0.0.0:6052 and configuration dir ....

Browsing to http://localhost:6052 yielded me the following page:

Creating a Firmware #

I proceeded to create a new device, and entered some basic information about my Wifi network:

This is where things got interesting (for me, at least!). It turns out that the ESPHome dashboard makes use of the Web Serial API to program the ESP32 chip over USB from the browser. Given that my board was already connected and in programming mode, this was pretty simple!

The first time I tried this, I got a permission error. After looking through the messages in my kernel’s ring buffer with journalctl -k, and looking at the permissions on the /dev/ttyUSB0 device, it seemed likely that the issue was the root:root ownership with limited permissions. I solved this by running the following:

1
2


sudo chown root:users /dev/ttyUSB0
sudo chmod g+rwx /dev/ttyUSB0

This changed the group of the serial device to the same group as my user, then gave the group read, write and execute permissions. I retried in the browser and the device was flashed with the new firmware, connecting it to my network! Once the device was online, it appeared as such in the ESPHome dashboard, and I cooked up the following configuration by studing the docs on the inkbird_ibsth1_mini sensor component:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44


substitutions:
 name: esphome-web-caf3b8
 friendly_name: Tub Monitor

esphome:
 name: ${name}
 friendly_name: ${friendly_name}
 min_version: 2024.6.0
 name_add_mac_suffix: false
 project:
 name: esphome.web
 version: dev

esp32:
 board: esp32dev
 framework:
 type: arduino

web_server:
improv_serial:
logger:
api:
ota:
 - platform: esphome

wifi:
 # Pulled in from a separate secrets.yaml
 ssid: !secret wifi_ssid
 password: !secret wifi_password

esp32_ble_tracker:
 scan_parameters:
 active: true
 continuous: true
 duration: 1min

sensor:
 - platform: inkbird_ibsth1_mini
 # Bluetooth MAC address of my IBS-P01B
 mac_address: <redacted>
 temperature:
 name: "Tub Temperature"
 battery_level:
 name: "Tub Monitor Battery"

A few points on the above:

The device setup wizard created a secrets.yaml which can hold secrets to be referenced with the !secret <secret name> syntax. This is the same format as used by Home Assistant.
Specifying web_server: enables a small embedded web server that can be used to check status (see below!).
I had to specify the MAC address of the pool thermometer, which I collected using the Bluetooth Inspector app for iOS.

I clicked Install in the top right of the window, and the firmware was compiled and flashed to the device! This time I was able to flash over Wifi since the device was now connected to my network as a result of the onboarding process:

After selecting “Wirelessly”, and waiting for the firmware to be flashed, the logs shortly started flowing in:

To double check things were working, I then browsed to the IP address of the ESPHome device, and was greeted with a simple page that shows the current state of the device:

Adding to Home Assistant was as simple as adding a new integration of type ESPHome and specifying the IP address of the device!

3D Printing an Enclosure #

The final part of my build was to create a small enclosure for the device that would house the antenna correctly. I found a nice model on Printables, which printed pretty fast on my BambuLab X1 Carbon printer.

The photo below shows the device in place on the window sill, with the hot tub just visible in the background:

Home Assistant Dashboard #

With all this in place, all that remained was tying the information into my Home Assistant dashboard:

The above shows the hot tub temperature, as well as the battery level of the IBS-P01 sensor. Beneath, I’ve added basic controls and measurements available through the Tapo Home Assistant integration, which allows me to quickly toggle the pump and heater, and see their daily energy usage at a glance.

If you look closely, you’ll see the room temperatures resulting from the Roth underfloor heating integration I wrote about previously.

Summary #

We’re super pleased with our tub - it was built by Andy from Rustic Tubs. If you’re considering one and you’re in the UK, you could do a lot worse. It’s beautifully crafted, and Andy was really helpful and communicative throughout the process.

This was my first foray into ESP32-based projects, and I was pleasantly surprised with how seamless the process was. The ESPHome docs were clear, and the no-hassle flashing through the browser was a nice way to get started.

I hope this post is useful to people who are just getting started, and feel free to reach out if I’ve missed something!

Writing a Home Assistant Core Integration: Part 2

Wed, 16 Oct 2024 00:00:00 +0000

Introduction #

In my last post I described the first steps toward creating a new Home Assistant integration for the underfloor heating system in my house. In that post I outlined in detail how I set about creating a Python client for the API provided by the underfloor heating controller vendor.

In this post, I’ll describe the development setup, project structure and contribution process for building and landing a Home Assistant Core integration. I don’t consider myself an expert here, but I’ve documented my journey here in the hope that my experience might be useful to potential future contributors.

The finished integration can be seen in the Home Assistant docs under the name touchline_sl, and the code can be found on Github.

Development Setup #

The Home Assistant documentation recommends the use of Visual Studio Code with a devcontainer. This was a very quick way to get a working environment up and running, especially given that I already use Visual Studio Code for most of my programming, so I was immediately familiar.

The repository provides some tasks to help get started, including a task named Run Home Assistant Core, which takes care of setting up the runtime environment, installing dependencies and starting the server. Neat!

There are also a set of pre-commit hooks set up to ensure you don’t make any common mistakes, accidentally violate the formatting/static-typing rules for the project, forget to update requirements.txt files, etc.

This turned out to be a really nice way to get started, and if you’re new to Home Assistant Core development, I’d recommend giving this “batteries-included” approach a go. If it’s not for you, the project provides manual setup instructions too.

Integration Basics #

According to the documentation:

[An] integration is responsible for a specific domain within Home Assistant. Integrations can listen for or trigger events, offer actions, and maintain states.

Where a domain is…

a short name consisting of characters and underscores. This domain has to be unique and cannot be changed.

Err, right… while this is an accurate statement, it’s perhaps not the most enlightening for the budding new integration developer! At their core, integrations are Python modules that take information about a given system (like an underfloor heating system, or a smart plug, or a light bulb) and represent information about them in a format compatible with one of Home Assistant’s archetypes for devices/sensors/platforms/entities.

In this case, we’ll be representing Climate Entities, which have the sort of properties you might expect - target_temperature, current_humidity, current_temperature, etc.

To use Home Assistant’s terms, in my setup:

The platform is the underfloor heating system
The platform consists of some devices, in this case the physical thermostats in each room of my house
The devices each represent one or more climate entities (humidity, temperature, etc.)

File Structure #

The basic file structure can be laid down with some scaffold tooling, but even in its finished state, my integration doesn’t have many files:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


.
├── __init__.py # the component file
├── climate.py # ties info from the api into home assistant terms
├── config_flow.py # defines the fields/flow for integration config
├── const.py # constants used across the integration
├── coordinator.py # data update coordinator
├── manifest.json # defines project dependencies and metadata
├── strings.json # defines strings displayed in various ui elements
└── translations # a directory containing one file per language
 └── en.json # english translation of strings.json

And to give an idea of the scale of the project in its completed form:

----------------------------------------------------------------------
Language files blank comment code
----------------------------------------------------------------------
Python 5 71 27 215
JSON 3 0 0 82
----------------------------------------------------------------------
SUM: 8 71 27 297
----------------------------------------------------------------------

`manifest.json` #

Starting with the most simple first! The manifest.json describes the integration: what its name is, where its documentation is found, who owns the code and the libraries it depends on:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


{
 "domain": "touchline_sl",
 "name": "Roth Touchline SL",
 "codeowners": ["@jnsgruk"],
 "config_flow": true,
 "documentation": "https://www.home-assistant.io/integrations/touchline_sl",
 "integration_type": "hub",
 "iot_class": "cloud_polling",
 "requirements": ["pytouchlinesl==0.1.8"]
}

Here I selected cloud_polling as the iot_class, because my integration reaches out periodically to the API, polling for new information. You’ll note also that the integration requires my pytouchlinesl library in order to run.

`init.py` #

Typically for integrations, this file defines how to setup the integration, and how to unload it. You can see the full source of my implementation on Github.

I define an async_setup_entry() method which takes care of:

Establishing a coordinator per TouchlineSL module
Performing the initial hydration of data
Registering each TouchlineSL module as a device in Home Assistant’s Device Registry, which is where Home Assistant “keeps track of devices”

There is a shortened, annotated version of the method below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


async def async_setup_entry(hass: HomeAssistant, entry: TouchlineSLConfigEntry) -> bool:
 """Set up Roth Touchline SL from a config entry."""
 account = TouchlineSL(...)

 coordinators: list[TouchlineSLModuleCoordinator] = [
 TouchlineSLModuleCoordinator(hass, module)
 for module in await account.modules()
 ]
 # ...
 device_registry = dr.async_get(hass)
 # Create a new Device for each coorodinator to represent each module
 for c in coordinators:
 module = c.data.module
 device_registry.async_get_or_create(
 config_entry_id=entry.entry_id,
 identifiers={(DOMAIN, module.id)},
 name=module.name,
 # ...
 )

 entry.runtime_data = coordinators
 await hass.config_entries.async_forward_entry_setups(entry, PLATFORMS)
 return True

Before returning async_setup_entry() invokes async_forward_entry_setups(). This ensures that async_setup_entry() in climate.py is called to ensure each of the climate entities is registered.

The PLATFORMS variable is a list of platform types that the integration supports, in this case a single-item list containing just Platform.CLIMATE, which is how async_forward_entry_setups knows to invoke the async_setup_entry() method in climate.py:

1
2
3
4


from homeassistant.const import Platform
# ...
PLATFORMS: list[Platform] = [Platform.CLIMATE]
# ...

To initialise the integration, the user must authenticate with the Roth API so that module details are fetched before constructing the coordinator. Thus, before this code is executed, the user must go through the config flow…

`config_flow.py` #

Despite the final implementation being quite simple, this is probably one of the areas I found most challenging to get right. There are some docs but they only scratch the surface, and implementations in other integrations seem to vary quite dramatically (mostly depending on when they were written).

I went through a few iterations of this config flow, mostly because I had originally implemented the ability to select a specific module from the user’s account. The review process guided me toward simply logging into the account, then enrolling each of the modules associated with the account - since users can always disable entities they don’t wish to manage in Home Assistant.

The config_flow.py defines which input fields need to be presented to the user, and then passes on the relevant information needed to set up the integration. In my implementation, the code:

Prompts the user for their username and password
Authenticates with the service and fetches the user’s unique ID
Registers that unique ID, aborting if the specified account has already been used
Creates a config entry in Home Assistant that stores the user’s credentials

The config flow also has some basic error handling that can distinguish the difference between poor credentials, networking issues, etc. The full implementation can be seen on Github, but the important parts are highlighted in the following snippet:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


# ...
class TouchlineSLConfigFlow(ConfigFlow, domain=DOMAIN):
 # ...
 async def async_step_user(
 self, user_input: dict[str, Any] | None = None
 ) -> ConfigFlowResult:
 # ...
 if user_input is not None:
 try:
 account = TouchlineSL(
 username=user_input[CONF_USERNAME],
 password=user_input[CONF_PASSWORD],
 )
 # Use the credentials to fetch unique user id
 await account.user_id()
 except RothAPIError as e:
 # ...
 else:
 # Set unique ID, abort setup if already used
 unique_account_id = await account.user_id()
 await self.async_set_unique_id(str(unique_account_id))
 self._abort_if_unique_id_configured()

 # Create a config entry containing the user's credentials
 return self.async_create_entry(
 title=user_input[CONF_USERNAME], data=user_input
 )

 return self.async_show_form(
 step_id="user", data_schema=STEP_USER_DATA_SCHEMA, errors=errors
 )

What caught me out was how the fields are given titles/descriptions. These attributes are all configured in the strings.json, where the config map contains keys for each of the config “steps”.

The code above defines a step named user, since the method name is async_step_user. The step’s name, description and input fields are defined in the strings.json:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


{
 "config": {
 "flow_title": "Touchline SL Setup Flow",
 "error": {
 "cannot_connect": "[%key:common::config_flow::error::cannot_connect%]"
 // ...
 },
 "step": {
 "user": {
 "title": "Login to Touchline SL",
 "description": "Your credentials for the Roth Touchline SL mobile app/web service",
 "data": {
 "username": "[%key:common::config_flow::data::username%]",
 "password": "[%key:common::config_flow::data::password%]"
 }
 }
 },
 "abort": {
 "already_configured": "[%key:common::config_flow::abort::already_configured_device%]"
 }
 }
 // ...
}

The values displayed to the user are pulled from the translation files at runtime depending on their language configuration. In my integration, the corresponding translations/en.json contains the following fields that map to those defined above:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


{
 "config": {
 "flow_title": "Touchline SL Setup Flow",
 "error": {
 "cannot_connect": "Failed to connect"
 // ...
 },
 "step": {
 "user": {
 "data": {
 "password": "Password",
 "username": "Username"
 },
 "description": "Your credentials for the Roth Touchline SL mobile app/web service",
 "title": "Login to Touchline SL"
 }
 },
 "abort": {
 "already_configured": "Device is already configured"
 }
 }
 // ...
}

In hindsight, this is covered in the docs, but it definitely didn’t click with me when I was going through it, so it seemed worth calling out! The net result of this setup is a configuration dialog that looks like so:

`coordinator.py` #

This was an addition I made during the review process (more on that later), and appears to be the preferred way to implement the fetching of data from upstream APIs. By implementing a DataUpdateCoordinator, Home Assistant can ensure that a single coordinated poll happens across all entities managed by an integration. If an integration manages many entities for which it needs to fetch/update details, the coordinator helps ensure that the API is called only as often as is needed.

The coordinator class is very simple: it defines a single method _async_update_data which returns the data for the device its coordinating. As the developer, you can specify the type of the data returned by the coordinator. I chose to represent this as a Python dataclass:

1
2
3
4
5


@dataclass
class TouchlineSLModuleData:
 module: Module
 zones: dict[int, Zone]
 schedules: dict[str, GlobalScheduleModel]

The coordinator is initialised with some basic information such as a name and an update interval:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# ...
class TouchlineSLModuleCoordinator(DataUpdateCoordinator[TouchlineSLModuleData]):
 def __init__(self, hass: HomeAssistant, module: Module) -> None:
 super().__init__(
 hass,
 logger=_LOGGER,
 name=f"Touchline SL ({module.name})",
 update_interval=timedelta(seconds=30),
 )

 self.module = module

The _async_update_data method then queries the API, and returns data in the newly defined format:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


async def _async_update_data(self) -> TouchlineSLModuleData:
 try:
 zones = await self.module.zones()
 schedules = await self.module.schedules()
 except RothAPIError as error:
 # ...
 # Return the data using our dataclass from above
 return TouchlineSLModuleData(
 module=self.module,
 zones={z.id: z for z in zones},
 schedules={s.name: s for s in schedules},
 )

You can see the full implementation on Github.

`climate.py` #

And finally, on to the business logic of tying fields from the upstream API into the relevant attributes in Home Assistant!

The first task handled by this file is registering each of the climate entities by iterating over each zone, in each coordinator’s module:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# ...
async def async_setup_entry(
 hass: HomeAssistant,
 entry: TouchlineSLConfigEntry,
 async_add_entities: AddEntitiesCallback,
) -> None:
 """Set up the Touchline devices."""
 coordinators = entry.runtime_data
 async_add_entities(
 TouchlineSLZone(coordinator=coordinator, zone_id=zone_id)
 for coordinator in coordinators
 for zone_id in coordinator.data.zones
 )
# ...

Home Assistant entities have well-defined APIs - the docs for climate entities show the supported attributes and their data types. I used a combination of the docs, and the source code to establish how to implement my ClimateEntity, which boiled down to the following interface:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


class TouchlineSLZone(CoordinatorEntity[TouchlineSLModuleCoordinator], ClimateEntity):
 # Construct a Touchline SL climate zone.
 def __init__(self, coordinator: TouchlineSLModuleCoordinator, zone_id: int) -> None:
 # Handle updated data from the coordinator.
 @callback
 def _handle_coordinator_update(self) -> None:
 # Return the device object from the coordinator data.
 @property
 def zone(self) -> Zone
 # Report if the device is available.
 @property
 def available(self) -> bool
 #Set new target temperature.
 async def async_set_temperature(self, **kwargs: Any) -> None
 # Assign the zone to a particular global schedule.
 async def async_set_preset_mode(self, preset_mode: str) -> None
 # Populate attributes with data from the coordinator.
 def set_attr(self) -> None:

Arguably the most important part here is set_attr(). which takes care of mapping fields from the objects provided by my pytouchlinesl library to attributes on the climate entity:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


def set_attr(self) -> None:
 """Populate attributes with data from the coordinator."""
 schedule_names = self.coordinator.data.schedules.keys()

 self._attr_current_temperature = self.zone.temperature
 self._attr_target_temperature = self.zone.target_temperature
 self._attr_current_humidity = int(self.zone.humidity)
 self._attr_preset_modes = [*schedule_names, CONSTANT_TEMPERATURE]

 if self.zone.mode == "constantTemp":
 self._attr_preset_mode = CONSTANT_TEMPERATURE
 elif self.zone.mode == "globalSchedule":
 schedule = self.zone.schedule
 self._attr_preset_mode = schedule.name

The full implementation is on Github.

Testing #

Testing is (understandably) a little complicated. The requirements for landing code for an integration stipulate that you must include unit tests for any config/options flows. In my case, this meant the following:

Implementing a mock TouchlineSL client fixture to ensure that the unit tests don’t try to reach out to the real Roth API
Implementing a mock config entry fixture
A unit test for a successful config flow execution
A parametrised unit test for unsuccessful config flow due to possible exceptions when hitting the API
A unit test to ensure that multiple config flows resulting in the same user ID fail

To me this feels like the bare minimum, and unfortunately doesn’t really provide any confidence that the integration actually functions correctly. I’m hoping to improve this in the future, but for now further testing has been manual.

Docs & Brand #

As I was creating the Pull Request to contribute my integration, I was prompted by the template to link to further PRs that added the documentation and brand assets for my integration. I used the existing touchline docs as a template and modified them for my integration. The docs pull request added a single file touchline_sl.markdown containing 29 lines, which results in some nicely rendered docs on the Home Assistant website:

There were already branding assets for Roth as part of the touchlinesl integration that was previously merged. Based on some review feedback, I created a new Integration Brand named roth, with which both the touchline and touchline_sl integrations are associated. This has the nice effect of grouping them when setting up a new integration:

Contribution Process #

I submitted my first efforts for review in home-assistant/core#124557. The checklist guided me nicely through what needed to be done, and overall the process went pretty smoothly, and pretty quickly (despite the 117 comments!). The process took a little under two days in total. I was also fortunate with my timing, since my code landed the day before the next beta release was cut, so it shipped relatively quickly.

I’d like to offer my sincere thanks to @joostlek who not only reviewed my code extremely quickly, but took the time to explain things to me both in the Github PR, but also by proactively reaching out to me on the Home Assistant Discord, which I really appreciated.

I got a lot of feedback, which I expected. This was my first attempt at writing code for Home Assistant, and it’s a pretty large and well established project. I do think the project could do with some better developer documentation, which would dramatically reduce the burden of effort on reviewers and give contributors a better chance of “getting it right”.

One might argue I could have done more reading and more research - though I spent a good amount of time reading both the documentation and the source code of other integrations. I’m not sure I’d ever have reached the conclusions that @joostlek kindly nudged me toward.

Overall, my implementation was more brief, more simple and more efficient as a result of the review process, and based on my experience I’d advocate for having a go if you’ve been on the fence! Often submitting code to such a project can be daunting, but as with my experience when contributing to nixpkgs, if you go in with an open mind you’re sure to learn something from the process.

Results #

I’m really pleased with the results. However obtuse the developer experience felt at times, there is no denying that what I was able to get from my 297 lines of implementation code is quite staggering. I was super impressed that just by following the conventions I was able to get such intuitive controls, a full graphable history of temperatures and such a wide variety of ways to display the information in my various dashboards.

Once set up, you’re able to get a view of all the different zones imported by the module (the names of each zone are pulled from the upstream API if the zones are named, and each can be associated with a given area in Home Assistant):

The default dashboard displays thermostat controls for each zone. These allow you to see the current and target temprature, as well as adjust the target temperature if you need to:

Expanding the thermostat cards gives you a more detailed view, showing the mode and the “Preset”. My implementation maps “Presets” to “Global Schedules” configured in the Roth module:

Summary #

And that’s a wrap! I learned a bunch writing this integration, and the resulting user experience is quite a lot better than I get with the default Roth application.

My next mission is to use the information from these climate entities to automate opening the Velux windows in the roof when things get warm in the summer, but I’ve got a lot to learn about Home Assistant in the mean time!

Writing a Home Assistant Core Integration: Part 1

Wed, 11 Sep 2024 00:00:00 +0000

Introduction #

Back in March, my family and I moved into a new home. It’s a modern construction which came with solar panels (and associated inverter/battery storage), and uses an air source heat pump to heat the house with underfloor heating. Being a new renovation, nearly all of the appliances and components in the house have a form of internet connectivity (some more useful than others!).

Since day 1, I’ve been hoping to consolidate all of the various applications, data feeds and functions into one single place. I’ve been a long-time listener to the Self Hosted podcast, which often extols the virtues of Home Assistant. I’ve got no prior experience with Home Assistant, but for the last three months I’ve been running it on my home server, with a collection of custom integrations and hacks that enable me to control the underfloor heating and solar inverter.

The underfloor heating controller is a Roth Touchline SL system. In my set up, there is a single “module” which represents my house, and a number of “zones” which represent different rooms.

There was unfortunately no integration for this system in Home Assistant - there is one for the previous generation “Roth Touchline”, but this appears to function over the LAN, whereas the Touchline SL system is controlled over the internet using their API.

After reading the source code of a few other climate integrations in Home Assistant, it became clear to me that the first step was to create a Python client for the API which could be used in the integration.

This post will cover the design, implementation and limitations of the library I wrote: pytouchlinesl. If you came here to read about writing code for Home Assistant, you’ll have to wait for the next post! 😉

Designing the library #

Upstream API #

Usually, one would interact with a Roth Touchline SL system through their mobile apps, or through their online portal. The mobile app seems to be quite a lightweight wrapper around the web application, and I’ve not been able to detect any difference in functionality.

A bit of searching uncovered that Roth also provide an API for the Touchline SL system, and an OpenAPI spec. This made the process significantly easier, though there are some discrepancies in what is documented compared with how the API actually behaves. It feels to me like the API may have evolved, and the documentation has remained static - or perhaps it was always inaccurate? Either way, I spent quite a bit of time manually fuzzing the API to work out the correct set of parameters for some endpoints.

I also studied the web application using the Chrome Dev Tools. Of all the endpoints documented, it seemed like I’d only need the following:

POST /authentication: authenticates with the API, taking a username and password, and returning a token
GET /users/{user_id}/modules: returns a list of modules associated with the user
GET /users/{user_id}/modules/{module_udid}: returns details of a specific module (zones, schedules, etc.)

There is a slight awkwardness here, in that the last endpoint returns all of the data - for all zones, all schedules, etc. This feels inefficient, but I couldn’t find a way of getting information about a specific zone, or a specific schedule. The web application seems to rely upon polling the (undocumented) endpoint GET /users/{user_id}/modules/{module_id}/update/data/parents/[]/alarm_ids/[]/last_update/{timestamp}, which delivers deltas in the data since a given timestamp. This is useful in the context of the app because it can request the full dataset once, then request only changes from that point onwards, keeping the app state up to date without requesting the whole dataset.

Making changes to the configuration of zones and their temperatures is also fragmented. In essence, one can:

Set a zone to a constant temperature: POST /users/{user_id}/modules/{module_udid}/zones
Place a zone on a global schedule: POST /users/{user_id}/modules/{module_udid}/zones/{zone_id}/global_schedule
Place a zone on a local schedule: POST /users/{user_id}/modules/{module_udid}/zones/{zone_id}/local_schedule

The first is self-explanatory, enabling a zone to be set to a constant temperature (19.0C, for example). Touchline SL modules also support “schedules” which contain time periods for the specified zones to reach certain temperatures. In the case of a “Global Schedule”, multiple zones can be assigned, while a “Local Schedule” is specific to a single zone. The awkwardness in the API here is that to “add” a zone to a global schedule, you must re-specify the entire schedule, and specify all of the zones that should be on the schedule…

Basic Requirements #

In order to fulfil the basic functionality of my (future) Home Assistant integration, I limited the requirements of the first version to:

Authenticate with the API using a username and password
List modules associated with the account
Get a specific module
Get a specific zone
Get a list of global schedules
Get a specific global schedule
Assign a constant temperature to a zone
Assign a zone to a specific global schedule

I don’t use local schedules in my system, so I’ve omitted them for now, though updating the library to support them would be trivial.

Outline design/experience #

With those requirements in mind, I came up with a rough sketch of how I’d like the library to behave:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


tsl = TouchlineSL(username="foo", password="bar")
module = await tsl.module(id="deadbeef")

lounge = module.zone_by_name("Lounge")
kitchen = module.zone(1234)

# Properties should be available such as:
# lounge.current_temperature
# kitchen.humidity

await kitchen.set_temperature(temperature=20.0)

living_spaces = await tsl.schedule_by_name(schedule_name="Living Spaces")
await lounge.set_schedule(living_spaces.id)

And from that came a reasonable outline of the API for the library:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


class TouchlineSL:
 # Construct a class that represents a Touchline SL account
 def __init__( self, *, username: str | None = None, password: str | None = None)
 # Get a list of modules associated with the account
 async def modules(self) -> list[Module]
 # Get a specific module, by ID
 async def module(self, *, module_id: str) -> Module | None

class Module:
 # Get a list of zones from the module, optionally including disabled zones
 async def zones(self, *, include_off: bool = False) -> list[Zone]
 # Get a specific zone, by ID
 async def zone(self, zone_id: int) -> Zone | None
 # Get a specific zone, by name
 async def zone_by_name(self, zone_name: str) -> Zone | None
 # Get a list of global schedules
 async def schedules(self) -> list[Schedule]:
 # Get a specific schedule, by ID
 async def schedule(self, schedule_id: int) -> Schedule | None
 # Get a specific schedule, by name
 async def schedule_by_name(self, schedule_name: int) -> Schedule | None

class Zone:
 # Get the schedule the zone is assigned to
 def schedule(self) -> Schedule | None
 # Set the zone to a constant temperature
 async def set_temperature(self, temperature: float)
 # Assign the zone to a specific schedule
 async def set_schedule(self, schedule_id: int)

Note that after reading the code from other climate integrations in Home Assistant, it became clear to me that they favour the use of async libraries, and thus my library was designed to use asyncio from the start.

Python tools/libraries #

There are a couple of things I’ve found tiresome about Python over the years, but things do seem to be looking up. I’ve always found the package management and distribution to be awkward, and I can’t be the only one if the number of projects looking to target that problem is anything to go by (e.g. poetry, rye, uv, pdm, etc.).

Part of this seems to come from fractures in the community itself - there (still!) appears to be disagreements surrounding PEPs such as PEP-621 which introduced pyproject.toml as a way of managing project metadata and dependencies, with the maintainers of some high-profile and widely adopted libraries refusing to adopt it.

That said, there are a couple of things I’ve been meaning to try in anger, and this project was a good opportunity to do so:

`uv` #

Developed by Astral, uv is the “new shiny” at the time of writing, and I can understand why. Pitched as “Cargo, but for Python”, it aims to solve a myriad of problems in the Python ecosystem. uv can handle the download/install of multiple Python versions, the creation of virtual environments, running Python tools in a one-off fashion (like pipx), locking dependencies deeply in a project (by hash) and still maintains a pip compatible command-line experience with uv pip. To add to all of that, it’s ridiculously fast; on a couple of occasions I’ve actually found myself wondering if it did anything when installing dependencies for large projects, because it’s so much faster than I’m used to.

`pydantic` #

Pydantic is a data validation library for Python. It’s entirely driven by Python’s type-hints which means that you get nice integration with language servers. Pydantic allows you define data models in native Python, but emit standard JSON Schema docs for models. It’s integrated quite widely across the Python ecosystem, and to me feels like it bridges the gap between what I hoped type annotations would do for Python, and what they actually do in reality!

`ruff` #

I’ve been using this one for at least the last year for all my Python linting and formatting needs, but I still feel it deserves a call out. There have been a couple of small changes to command line API and such along the way, but overall I’ve found it to be a dramatic improvement over my last setup - which comprised of black, isort, flake8 and a pile of plugins. I had no particular beef with black, but I find flake8’s lack of pyproject.toml support irritating, and grew tired of plugins failing as flake8 released new versions.

In my experience, ruff is stupid fast, and because it ships flake8-compatible rules for all of the plugins I was using as one bundle, they never break. It’s also nice to just have one tool to use everywhere. If you’re interested, you can see how I configure ruff in the pyproject.toml.

Implementation #

With a basic design in mind, and tooling ready to go, I set about building the library itself. It was now time to reconcile my intended design with the realities of the provisions made by the upstream API.

I mentioned previously that the only useful endpoint for getting information about zones/schedules would in fact return all of the data for a given module. Too many calls to this endpoint would likely result in poor performance, so I wanted to introduce some basic caching along the way.

Client implementation #

For the underlying API client implementation, I opted for the following:

BaseClient: a class which inherits from Python’s abc.ABC. This enables the creation of multiple client implementations by defining of the set of methods/properties that any client interacting with the Roth API should define. This decision is primarily to support testing through dependency injection, rather than mocking with patches (more details on that later).
RothAPI: a concrete implementation of the BaseClient abstract class. It is here that I built the actual implementation of the client which handles authentication, GETing and POSTing data, caching, and marshalling API responses into the correct types (defined with Pydantic).

Also included in the client package is the models package. The models package contains (mostly) auto-generated Pydantic models based on real-life responses I got from the API. This is really a function of laziness, but was a convenient way to get type annotated models for the responses I was receiving from the API. Each time I hit a new endpoint, I took the JSON result and did a quick conversion with https://jsontopydantic.com/, before manually adjusting names and updating some fields with Literals.

Caching #

I mentioned earlier that I wanted to implement some basic caching. While I am aware of various plugins for aiohttp (and other request libraries) that could handle this for me, my requirements were quite simple, so I chose to just build it into the library. In this case, caching is implemented on the Module class. This is because the large blob of data that is requested to populate details about a module and its zones/schedules is requested per module.

The caching works like this:

The Module class has “private” attributes named _raw_data and _last_fetched:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


class Module:
 def __init__(
 self,
 *,
 client: BaseClient,
 module_data: AccountModuleModel,
 cache_validity: int = 30,
 ):
 # ...

 # Raw data about the zones, schedules, tiles in the module
 self._raw_data: ModuleModel
 # Unix timestamp representing the last time the _raw_data was fetched
 self._last_fetched = 0
 self._cache_validity = cache_validity

 # ...

There is only one method on this class that calls the underlying client, and that’s another “private” method named _data. This method takes an optional refresh keyword argument, which forces the _raw_data attribute to be updated, but by default will only fetch data if the cached data has expired (after the number of seconds specified in self._cache_validity). If refresh is false, and the cache hasn’t expired, it simply returns the stored raw data:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


async def _data(self, *, refresh: bool = False) -> ModuleModel:
 """Get the raw representation of the module from the upstream API.

 If the data has never been fetched from upstream, or the data is older
 than the cache validity period, then the data is refreshed using the
 upstream API.

 Args:
 refresh: (Optional): Force the data to be refreshed using the API.
 """
 if refresh or (round(time.time() * 1000) - self._last_fetched) > self._cache_validity:
 self._raw_data = await self._client.module(self.id)
 self._last_fetched = round(time.time() * 1000)

 return self._raw_data

Each of the public methods (zones(), zone(), schedule(), etc.) access the raw data through the _data() method, and pass through the refresh flag which is exposed. This means that any developer consuming this library can chose to live with the caching, or override it and force a refresh like so:

1
2
3
4
5
6
7


tsl = TouchlineSL(username="foo", password="bar")
module = await tsl.module(id="deadbeef")

# Request a zone, accepting cached data (default)
zone = await module.zone_by_name("kitchen")
# Or force the data to be refreshed using the upstream API
zone = await module.zone_by_name("kitchen", refresh=True)

Testing, CI & Publishing #

Testing #

In the previous section, I described how I’d set up an abstract base class for the API client, then created an implementation of that for the actual Roth API. One of the main reasons I like this pattern for API clients is that it simplifies testing and reduces the need for monkey patching or traditional mocking (yes, I know the fake API client is sort of a mock…).

Because TouchlineSL can optionally be constructed with any client that implements BaseClient’s abstract base class, it’s trivial to implement a fake API backend that returns fixture data to be used in testing. In this case, the fixtures are stored as JSON files in the repository, and contain real life responses received from the API.

The FakeRothAPI returns the sample data for each of the methods defined in the abstract class. The following is a partial extract from the fake client code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


# ...
from pytouchlinesl.client import BaseClient
from pytouchlinesl.client.models import AccountModuleModel, ModuleModel

data_dir = Path(os.path.realpath(__file__)).parent / "sample-data"


class FakeRothAPI(BaseClient):
 def __init__(self):
 self._user_id = 123456789
 self._token = "deadbeef"

 async def user_id(self) -> int:
 return self._user_id

 async def authenticated(self) -> bool:
 return True if self._token else False

 async def modules(self) -> list[AccountModuleModel]:
 with open(data_dir / "modules.json", "r") as f:
 data = json.load(f)

 return [AccountModuleModel(**m) for m in data]

#...

From there, I defined a number of test fixtures using pytest’s @pytest.fixture decorator, which provide an initialised TouchlineSL instance, backed by the fake client, a Module instance, and a Zone instance.

From there, the tests are fairly simple. Beyond injecting the fake client, there is no mocking required, which in my opinion keeps the test code much easier to read and understand. It also means I could focus more of my energy on validating the logic I was testing, rather than worrying about how patching might interact with the rest of the code. I’ve felt bad about mocking for years, and I think the clearest articulation I’ve seen is “Don’t Mock What You Don’t Own” in 5 Minutes.

One aspect of the test suite I don’t love is the use of time.sleep in certain tests. This is because my caching implementation relies on reading a timestamp to decide on whether it should refresh data. In general I steer away from sleeps in tests, as they’re often used to mask an underlying non-determinism, but in this case it felt a reasonable trade-off, given that I’m testing a time-based functionality.

CI #

I wanted to ensure that any pull requests were tested, and that they conform to the project’s formatting/linting rules. Since the project is hosted on Github, I used Github Actions for this. The pipeline for this project is pretty simple, it lints and formats the code with ruff failing if any files were changed by the formatter or any of the linting rules were violated.

Finally, uv is used to run pytest across a matrix of supported Python versions. I could have used uv to handle the download and install of different Python versions too, but the setup-python actions has served me perfectly well in the past.

Publishing #

Publishing is taken care of in CI. I don’t like having to remember the magic incantation for building, authenticating and publishing locally. I want the process to be as consistent and as transparent as possible for the people consuming the project.

What was new to me this time was publishing to PyPI with a “Trusted Publisher” setup. To quote their docs:

“Trusted publishing” is our term for using the OpenID Connect (OIDC) standard to exchange short-lived identity tokens between a trusted third-party service and PyPI. This method can be used in automated environments and eliminates the need to use manually generated API tokens to authenticate with PyPI when publishing.

Rather than setting up Github Actions to hold an API token in a Secret, there is some automation that links a Github project to a PyPI project. The project doesn’t even have to be previously published on PyPI to get started, and new projects can be configured as “Pending”, then published to for the first time from your CI system of choice 🚀.

I configured Github Actions to trigger the release of pytouchlinesl on new tags being pushed:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


# ...
publish:
 name: Build and Publish to PyPI
 runs-on: ubuntu-latest
 needs:
 - tests

 # Trusted publisher setup for PyPI
 environment:
 name: pypi
 url: https://pypi.org/p/pytouchlinesl
 permissions:
 id-token: write

 steps:
 - name: Checkout the code
 uses: actions/checkout@v4

 - name: Install `uv`
 run: |
 curl -LsSf https://astral.sh/uv/install.sh | sh

 - name: Build the package
 run: |
 uvx --from build pyproject-build --installer uv

 - name: Publish to PyPi
 uses: pypa/gh-action-pypi-publish@v1.10.0

Contributing to `nixpkgs` #

Finally, my Home Assistant server runs NixOS, so in order for the integration to be packaged easily in the future, I created a small PR to include pytouchlinesl in nixpkgs. The original PR went through pretty quickly - thanks to @drupol for the fast review!

Since then I’ve made a couple of minor version bumps to the library as I discovered small issues when building the integration, but the final derivation is quite compact (see below). The Python build tooling in Nix is quite mature at this point, and I gained a fair bit of experience using it when packaging snapcraft and charmcraft.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32


#...

buildPythonPackage rec {
 pname = "pytouchlinesl";
 version = "0.1.5";
 pyproject = true;

 disabled = pythonOlder "3.10";

 src = fetchFromGitHub {
 owner = "jnsgruk";
 repo = "pytouchlinesl";
 rev = "refs/tags/${version}";
 hash = "sha256-kdLMuxA1Ig85mH7s9rlmVjEsItXxRlDA1JTFasnJogg=";
 };

 build-system = [ setuptools ];

 dependencies = [
 aiohttp
 pydantic
 ];

 nativeCheckInputs = [
 pytestCheckHook
 pytest-asyncio
 ];

 pythonImportsCheck = [ "pytouchlinesl" ];

 # ...
}

Summary #

And that concludes the first part of this series! Hopefully you found this useful - I’ve learned a lot from people over the years by understanding how they approach problems, so I thought I’d post my methodology here in case it helps anyone refine their process.

I’m certainly no Python expert, so if you’ve spotted a mistake or you think I’m wrong, get in touch.

In the next post, I’ll cover writing the Home Assistant integration, and contributing it to Home Assistant.

Libations: Tailscale on the Rocks

Wed, 21 Aug 2024 00:00:00 +0000

Introduction #

I’m a long-time, self-professed connoisseur of cocktails. I’ve always enjoyed making (and drinking!) the classics, but I also like experimenting with new base spirits, techniques, bitters etc.

Over the years, I’ve collected recipes from a variety of sources. Some originated from books (such as Cocktail Codex and Cocktails Made Easy), others from websites (Difford’s Guide), and most importantly those that I’ve either guessed from things I’ve drunk elsewhere (like my favourite bar in Bristol, The Milk Thistle), or modifications to recipes from the referenced sources.

I wanted somewhere to store all these variations: somewhere that I could search easily from my iPhone (which is nearly always close to me when I’m making drinks). I wanted each recipe to fit in its entirety on my iPhone screen without the need to scroll.

Around the time I started thinking about this problem, I also learned about tsnet and was desperate for an excuse to try it out - and thus Libations was born as the product of two things I love: Tailscale and cocktails!

tswhat? #

Some time ago, Tailscale released a Go library named tsnet. To quote the website:

tsnet is a library that lets you embed Tailscale inside of a Go program

In this case, the embedded Tailscale works slightly different to how tailscaled works (by default, anyway…). Rather than using the universal TUN/TAP driver in the Linux kernel, tsnet instead uses a userspace TCP/IP networking stack, which enables the process embedding it to make direct connections to other devices on your tailnet as if it were “just another machine”. This makes it easy to embed, and drops the requirement for the process to be privileged enough to access /dev/tun.

One of the things I like about how tsnet presents applications as devices on the tailnet, is that you can employ ACLs to control who and what on your tailnet can access the application, rather than the device. I’ve solved this problem before by putting applications in their own systemd-nspawn container and joining those containers to my tailnet. Another nice option is tsnsrv which essentially acts as a Tailscale-aware proxy for individual applications, but in this case I wanted to bake it into the application - which I would only access over my tailnet.

Getting started with tsnet couldn’t be easier:

1
2
3


mkdir tsnet-app; cd tsnet-app
go mod init tsnet-app
go get tailscale.com/tsnet

That will get you set up with a basic Go project, with the tsnet library available. Create a new main.go file with the following contents:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


package main

import (
 "fmt"
 "log"
 "net/http"

 "tailscale.com/tsnet"
)

func main() {
 // Create a new tsnet server instance
 s := tsnet.Server{Hostname: "tsnet-test"}
 defer s.Close()

 // Have the tsnet server listen on :8080
 ln, err := s.Listen("tcp", ":8080")
 if err != nil {
 log.Fatal(err)
 }
 defer ln.Close()

 // Define a very simple handler with a simple Hello, World style message
 handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 fmt.Fprintf(w, "<html><body><h1>Hello from %s, tailnet!</h1>\n", s.Hostname)
 })

 // Start an HTTP server on the tsnet listener
 err = http.Serve(ln, handler)
 if err != nil {
 log.Fatal(err)
 }
}

This is about the most minimal example I could contrive. The code creates a simple instance of tsnet.Server with the hostname tsnet-app, listens on port 8080 and serves up a simple Hello, World! style message. On running the application you’ll see the following:

1
2
3
4
5


❯ go run .
2024/08/13 15:03:37 tsnet running state path /home/jon/.config/tsnet-tsnet-app/tailscaled.state
2024/08/13 15:03:37 tsnet starting with hostname "tsnet-test", varRoot "/home/jon/.config/tsnet-tsnet-app"
2024/08/13 15:03:38 LocalBackend state is NeedsLogin; running StartLoginInteractive...
2024/08/13 15:03:43 To start this tsnet server, restart with TS_AUTHKEY set, or go to: https://login.tailscale.com/a/deadbeeffeebdaed

Clicking the link will open a page in your browser that runs you through Tailscale’s authentication flow, after which you should be able to curl the page directly from any of your devices (assuming you’re not doing anything complicated with ACLs that might prevent it)!

1
2
3
4
5
6
7
8


❯ tailscale status
# ...
100.93.165.28 kara jnsgruk@ linux -
100.106.82.10 tsnet-test jnsgruk@ linux -
# ...

❯ curl http://tsnet-test:8080
<html><body><h1>Hello from tsnet-test, tailnet!</h1>

The library has a pretty small API surface, all of which is documented on pkg.go.dev.

Libations #

For my cocktail app, I wanted to employ a similar, albeit simplified, set of techniques that I use to build this blog. I love Go’s ability to embed static files that can be served as web assets.

Recipe Schema #

I wanted to represent recipes in a format that could be updated by hand if necessary, and easily parsed into a web frontend. I decided to use JSON for this, representing the recipes as a list:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


[
 {
 "id": 10,
 "name": "New York Sour",
 "base": ["Bourbon"],
 "glass": ["12oz Lowball"],
 "method": ["Dry Shake", "Shake"],
 "ice": ["Cubed"],
 "ingredients": [
 { "name": "Lemon Juice", "measure": "20", "unit": "ml" },
 { "name": "Sugar", "measure": "20", "unit": "ml" },
 { "name": "Red Wine", "measure": "10", "unit": "ml" },
 { "name": "Bourbon", "measure": "40", "unit": "ml" },
 { "name": "Egg White", "measure": "20", "unit": "ml" }
 ],
 "garnish": ["Lemon Sail"],
 "notes": "Use claret or malbec"
 }
]

This schema is able to capture all the relevant details from the different formats I’ve seen over the years. It would take some time to format my favourites into this schema, but that was always going to be the case.

I was fortunate enough to get access to the recipe collection from a well regarded cocktail bar in the UK. Unfortunately it was given to me in a hard-to-parse PDF, which resulted in many hours of playing with OCR tools and manual data cleaning - but enabled me to bootstrap the app with around 450 high quality recipes. I didn’t include their recipes in the libations repository, but I did include some of my own favourite concoctions in a sample recipe file. My Mezcal Margarita gets pretty good reviews 😉.

Server #

The server implementation needed to fulfil a few requirements:

Parse a specific recipes file, optionally passed via the command line
Have an embedded filesystem to contain static assets and templates
Be able to render HTML templates with the given recipes
Listen on either a tailnet (via tsnet), or locally (for testing convenience)
When listening on the tailnet, listen on HTTPS, redirecting HTTP traffic accordingly

I wanted to keep dependencies to a minimum to make things easier to maintain over time. The tsnet library pulls in a few indirect dependencies, but everything else Libations uses is in the Go standard library.

The recipes JSON schema is very simple, and is modelled with a couple of Go structs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


// Ingredient represents the name and quantity of a given ingredient in a recipe.
type Ingredient struct {
 Name string
 Measure string
 Unit string
}

// Drink represents all of the details for a given drink.
type Drink struct {
 ID int
 Name string
 Base []string
 Glass []string
 Method []string
 Ice []string
 Ingredients []Ingredient
 Garnish []string
 Notes string
}

The parseRecipes function checks whether or not the user passed the path to a specific recipe file, or whether it should default to parsing the sample recipes file. Once it’s determined the right file to parse, and validated its existence, it unmarshals the JSON using the Go standard library.

Users have the option of passing the -local flag when starting Libations, which bypasses tsnet completely and starts a local HTTP listener on the specific port. This makes for easier testing when iterating through changes to the web UI and other elements:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


// ...
addr := flag.String("addr", ":8080", "the address to listen on in the case of a local listener")
local := flag.Bool("local", false, "start on local addr; don't attach to a tailnet")

// ...
var listener *net.Listener
if *local {
 listener, err = localListener(*addr)
} else {
 listener, err = tailscaleListener(*hostname, *tsnetLogs)
}
if err != nil {
 slog.Error("failed to create listener", "error", err.Error())
 os.Exit(1)
}
//...

Setting up the tsnet server and listener is only mildly more complicated – but mostly due to my requirement that all HTTP traffic is redirected to HTTPS, using the LetsEncrypt certificates that Tailscale provides automatically. The redirects are handled by a separate Goroutine in this case:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


// Start a standard HTTP server in the background to redirect HTTP -> HTTPS.
go func() {
 httpLn, err := tsnetServer.Listen("tcp", ":80")
 if err != nil {
 slog.Error("unable to start HTTP listener, redirects from http->https will not work")
 return
 }

 slog.Info(fmt.Sprintf("started HTTP listener with tsnet at %s:80", hostname))

 err = http.Serve(httpLn, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 newURL := fmt.Sprintf("https://%s%s", r.Host, r.RequestURI)
 http.Redirect(w, r, newURL, http.StatusMovedPermanently)
 }))
 if err != nil {
 slog.Error("unable to start http server, redirects from http->https will not work")
 }
}()

With the correct listener selected, I create an http.ServeMux to handle routing to static assets and rendering templates, and pass that mux to the http.Serve method from the Go standard library - and that’s it! At the time of writing the Go code totals 235 lines - not bad!

Web Interface #

The web interface was designed primarily for mobile devices, and I’ve not yet done the work to make it excellent for larger-screened devices - though it’s certainly bearable. It’s also read-only at the moment: you can browse all the recipes, and there is a simple full-text search which can narrow the list of recipes down by searching for an ingredient, method, glass type, notes, etc.

As mentioned in an earlier post, I’m a big fan of the Vanilla Framework, which is a “simple, extensible CSS framework” from Canonical, and is used for all of Canonical’s apps and websites. Given that I had some prior experience using it, I decided to use it again here. I started using my tried and tested recipe of Vanilla + Hugo, but later reverted to using simple HTML templates with Go’s html/template package.

The result is a set of templates, which iterate over the recipe data from the JSON file, and output nicely styled HTML elements:

There is nothing fancy going on here - it’s nearly all stock Vanilla Framework. I do specify some overrides to make the colours a bit less Ubuntu-ish, but that’s it!

One detail I’m pleased with is the dynamic drink icons. The icons indicate the type of glass the particular drink should be served in. This is a simple trick: for each drink the HTML template renders a glass-icon partial, which reads the glass type specified in the recipe, and renders the appropriate SVG file which is then coloured with CSS.

Packaging for NixOS #

There were two main tasks in this category: creating the Nix package itself, and writing a simple NixOS module that would make it simple for me to run it on my NixOS server.

The project uses a Flake to provide the package, overlay and module. The standard library in Nix has good tooling for Go applications now, meaning the derivation is short:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


{
 buildGo122Module,
 lastModifiedDate,
 lib,
 ...
}:

let
 version = builtins.substring 0 8 lastModifiedDate;
in
buildGo122Module {
 pname = "libations";
 inherit version;
 src = lib.cleanSource ../.;
 vendorHash = "sha256-AWvaHyJL7Cm+zCY/vTuTAsgLbVy6WUNfmaGbyQOzMMQ=";
}

I haven’t cut any versioned releases of Libations at the time of writing - I’m using the last modified date of the flake to version the binaries.

The module starts the application with systemd, and optionally provides it with a recipes file. There are four options defined at the time of writing: services.libations.{enable,recipesFile,tailscaleKeyFile,package}:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


options = {
 services.libations = {
 enable = mkEnableOption "Enables the libations service";

 recipesFile = mkOption {
 type = nullOr path;
 default = null;
 example = "/var/lib/libations/recipes.json";
 description = ''
 A file containing drinks recipes per the Libations file format.
 See https://github.com/jnsgruk/libations.
 '';
 };

 tailscaleKeyFile = mkOption {
 type = nullOr path;
 default = null;
 example = "/run/agenix/libations-tsauthkey";
 description = ''
 A file containing a key for Libations to join a Tailscale network.
 See https://tailscale.com/kb/1085/auth-keys/.
 '';
 };

 package = mkPackageOption pkgs "libations" { };
 };
};

The tailscaleKeyFile option enables the service to automatically join a tailnet using an API key, rather than prompting the user to click a link and authorise manually.

These options are translated into a simple systemd unit:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


config = mkIf cfg.enable {
 systemd.services.libations = {
 description = "Libations cocktail recipe viewer";
 wantedBy = [ "multi-user.target" ];
 after = [ "network.target" ];
 environment = {
 "XDG_CONFIG_HOME" = "/var/lib/libations/";
 };
 serviceConfig = {
 DynamicUser = true;
 ExecStart = "${cfg.package}/bin/libations -recipes-file ${cfg.recipesFile}";
 Restart = "always";
 EnvironmentFile = cfg.tailscaleKeyFile;
 StateDirectory = "libations";
 StateDirectoryMode = "0750";
 };
 };
}

The XDG_CONFIG_HOME variable is set so that tsnet stores it’s state in /var/lib/libations, rather than trying to store it in the home directory of the dynamically created user for the systemd unit.

I use agenix on my NixOS machines to manage encrypted secrets, and for this project I used it to encrypt both the initial Tailscale auth key, and my super-secret recipe collection! The configuration to provide the secrets and configure the server to run libations is available on Github, but looks like so:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


{ config, self, ... }:
{
 age.secrets = {
 libations-auth-key = {
 file = "${self}/secrets/thor-libations-tskey.age";
 owner = "root";
 group = "root";
 mode = "400";
 };

 libations-recipes = {
 file = "${self}/secrets/thor-libations-recipes.age";
 owner = "root";
 group = "root";
 mode = "444";
 };
 };

 services.libations = {
 enable = true;
 recipesFile = config.age.secrets.libations-recipes.path;
 tailscaleKeyFile = config.age.secrets.libations-auth-key.path;
 };
}

As a result, the application is now available at https://libations, with a valid LetsEncrypt certificate, on all of my machines! 🎉

Summary #

This was a really fun project. It felt like a fun way to explore tsnet, and resulted in something that I’ve used a lot over the past year. I don’t have many plans to adjust things in the near future - though I do find myself wanting a nice interface to add new recipes from time to time.

And now for the twist: three weeks ago I gave up drinking alcohol (likely for good), so now I’m on a mission to find some non-alcoholic recipes to make life a little tastier! I suspect this will be hard work - and I’ll certainly miss some of my favourites, but my wife and I have already found some compelling alternatives.

If you’ve got a favourite recipe (alcoholic or not) and you liked the article, then perhaps open a PR and add it to the sample recipes file!

Workstation VMs with LXD & Multipass

Tue, 25 Jun 2024 00:00:00 +0000

Introduction #

Over the years, I’ve used countless tools for creating virtual machines - often just for short periods of time when testing new software, trying out a new desktop environment, or creating a more isolated development environment. I’ve gone from just using the venerable qemu at the command line, to full-blown desktop applications like Virtualbox, to using virt-manager with libvirt.

When I joined Canonical back in March 2021, I’d hardly used LXD, and I hadn’t ever used Multipass. Since then, they’ve both become indispensable parts of my workflow, so I thought I’d share why I like them, and how I use each of them in my day to day work.

I work for Canonical, and am therefore invested in the success of their products, but at the time of writing I’m not responsible for either LXD or Multipass, and this post represents my honest opinions as a user of the products, and nothing more.

Installation / Distribution #

Both LXD and Multipass are available as snap packages, and that’s the most supported and recommended route for installation. LXD is available in the repos of a few other Linux distributions (including NixOS, Arch Linux), but the snap package also works great on Arch, Fedora, etc. I personally ran Multipass and LXD as snaps on Arch Linux for a couple of years without issue.

If you’d like to follow along with the commands in this post, you can get setup like so:

1
2
3
4
5
6
7
8
9


sudo snap install lxd
sudo lxd init --minimal

# If you'd like to use LXD/LXC commands without sudo
# run the following command and logout/login:
#
# sudo usermod -aG lxd $USER

sudo snap install multipass

Early on in my journey with NixOS, I packaged Multipass for Nix. I still maintain (and use!) the NixOS module. This was my first ever contribution to NixOS – a fairly colourful review process to say the least…

The result is that you can use something like the following in your configuration, and have multipass be available to you after a nixos-rebuild switch:

1
2
3


{
 virtualisation.multipass.enable = true;
}

LXD has been maintained in NixOS for many years now - and around this time last year I added support for the LXD UI. The screenshots you see throughout this post are all from LXD UI running on a NixOS machine using the following configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


{
 virtualisation.lxd = {
 enable = true;
 zfsSupport = true;
 ui.enable = true;
 };

 networking = {
 firewall = {
 trustedInterfaces = [ "lxdbr0" ];
 };
 };
}

Ubuntu on-demand with Multipass #

Multipass is designed to provide simple on-demand access to Ubuntu VMs from any workstation - whether that workstation is running Linux, macOS or Windows. It is designed to replicate, in a lightweight way, the experience of provisioning a simple Ubuntu VM on a cloud.

Multipass makes use of whichever the most appropriate hypervisor is on a given platform. On Linux, it can use QEMU, LXD or libvirt as backends, on Windows it can use Hyper-V or Virtualbox, and on macOS it can use QEMU or Virtualbox. Multipass refers to these backends as drivers.

Multipass’ scope is relatively limited, but in my opinion that’s what makes it so delightful to use. Once installed, the basic operation of Multipass couldn’t be simpler:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


❯ multipass shell
Launched: primary
Mounted '/home/jon' into 'primary:Home'
Welcome to Ubuntu 24.04 LTS (GNU/Linux 6.8.0-35-generic x86_64)

 * Documentation: https://help.ubuntu.com
 * Management: https://landscape.canonical.com
 * Support: https://ubuntu.com/pro

 System information as of Tue Jun 25 11:17:55 BST 2024

 System load: 0.4 Processes: 132
 Usage of /: 38.9% of 3.80GB Users logged in: 0
 Memory usage: 31% IPv4 address for ens3: 10.93.253.20
 Swap usage: 0%


Expanded Security Maintenance for Applications is not enabled.

3 updates can be applied immediately.
1 of these updates is a standard security update.
To see these additional updates run: apt list --upgradable

Enable ESM Apps to receive additional future security updates.
See https://ubuntu.com/esm or run: sudo pro status


ubuntu@primary:~$

This one command will take care of creating the primary instance if it doesn’t already exist, start the instance and drop you into a bash shell - normally in under a minute.

Multipass has a neat trick: it bundles a reverse SSHFS server that enables easy mounting of the host’s home directory into the VM. This happens by default for the primary instance. As a result the instance I created above has my home directory mounted at /home/ubuntu/Home - making it trivial to jump between editing code/files on my host and in the VM. I find this really useful - I can edit files on my workstation in my own editor, using my Yubikey to sign and push commits without having to worry about complicated provisioning or passthrough to the VM, and any files resulting from a build process on my workstation are instantly available in the VM for testing.

Multipass instances can be customised a little. You won’t find complicated features like PCI-passthrough, but basic parameters can be tweaked. The commands I usually run for setting up a development machine when I’m working on Juju/Charms are:

1
2
3
4
5
6


# Create a machine named 'dev' with 16 cores, 40GiB RAM and 100GiB disk
multipass launch noble -n dev -c 16 -m 40G -d 100G
# Mount my home directory into the VM
multipass mount /home/jon dev:/home/ubuntu/Home
# Get a shell in the VM
multipass shell dev

Once you’re done with an instance, you can remove it like so:

1
2


multipass remove dev
multipass purge

Multipass does have some more interesting features, though most of my usage is represented above. One feature that might be of more interest for MacOS or Windows users is aliases. This feature enables you to alias local commands to their counterparts in a Multipass VM, meaning for example that every time you run docker on your Mac, the command is actually executed inside the Multipass VM:

1
2
3


# Example of mapping the local `mdocker` command -> `docker` in
# the multipass VM
multipass alias dev:docker mdocker

Multipass will launch the latest Ubuntu LTS by default, but there are a number of other images available - including some “appliance” images for applications like Nextcloud, Mosquitto, etc.

There is also the concept of Blueprints which are essentially recipes for virtual machines with a given purpose. These are curated partly by the Multipass team, and partly by the community. A blueprint enables the author to specify cores, memory, disk, cloud-init data, aliases, health checks and more. The recipes themselves are maintained on Github, and you can see the list of available images/blueprints using multipass find:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


❯ multipass find
Image Aliases Version Description
core core16 20200818 Ubuntu Core 16
core18 20211124 Ubuntu Core 18
core20 20230119 Ubuntu Core 20
core22 20230717 Ubuntu Core 22
20.04 focal 20240612 Ubuntu 20.04 LTS
22.04 jammy 20240614 Ubuntu 22.04 LTS
23.10 mantic 20240619 Ubuntu 23.10
24.04 noble,lts 20240622 Ubuntu 24.04 LTS
daily:24.10 oracular,devel 20240622 Ubuntu 24.10
appliance:adguard-home 20200812 Ubuntu AdGuard Home Appliance
appliance:mosquitto 20200812 Ubuntu Mosquitto Appliance
appliance:nextcloud 20200812 Ubuntu Nextcloud Appliance
appliance:openhab 20200812 Ubuntu openHAB Home Appliance
appliance:plexmediaserver 20200812 Ubuntu Plex Media Server Appliance

Blueprint Aliases Version Description
anbox-cloud-appliance latest Anbox Cloud Appliance
charm-dev latest A development and testing environment for charmers
docker 0.4 A Docker environment with Portainer and related tools
jellyfin latest Jellyfin is a Free Software Media System that puts you in control of managing and streaming your media.
minikube latest minikube is local Kubernetes
ros-noetic 0.1 A development and testing environment for ROS Noetic.
ros2-humble 0.1 A development and testing environment for ROS 2 Humble.

The team also recently introduced the ability to snapshot virtual machines, though I must confess I’ve not tried it out in anger yet.

LXD… for VMs? #

For many people, LXD is a container manager - and indeed for many years it could “only” manage containers. LXD was built for running “system containers”, as opposed to “application containers” like Docker/Podman (or Kubernetes). Running a container with LXD is more similar to to running a container with systemd-nspawn, but with the added bonus that it can cluster across machines, authenticate against different identity backends, and manage more sophisticated storage.

Because LXD manages system containers, each container gets its own systemd, and behaves more like a ’lightweight VM’ sharing the host’s kernel. This turns out to be a very interesting property for people who want to get some of the benefits of containerisation (i.e. higher workload density, easier snapshotting, migration, etc.) with more legacy applications that might struggle to run effectively in application containers.

But this post is about virtual machines. Since the 4.0 LTS release, LXD has also supported running VMs with qemu. The API for launching a container is identical to launching a virtual machine. Better still, Canonical provides images for lots of different Linux distributions, and even desktop variants of some images - meaning you can quickly get up and running with a wide range of distributions, for example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# Launch a Ubuntu 24.04 LTS VM
lxc launch ubuntu:noble ubuntu --vm
# Get a shell inside the VM
lxc exec ubuntu bash

# Launch a Fedora 40 VM
lxc launch images:fedora/40 fedora --vm
# Get a shell inside the VM
lxc exec fedora bash

# Launch an Arch Linux VM (doesn't support secure boot yet)
lxc launch images:archlinux arch --vm -c security.secureboot=false
# Get a shell inside the VM
lxc exec arch bash

You can get a full list of virtual machine images like so:

1

lxc image ls images: --format=compact | grep VIRTUAL-MACHINE

LXD Desktop VMs #

Another neat trick for LXD is desktop virtual machines. These are launched with curated images that drop you into a minimal desktop environment that’s configured to automatically login. This has to be one of my favourite features of LXD!

1
2


# Launch a Ubuntu 24.04 LTS desktop VM and get a console
lxc launch images:ubuntu/24.04/desktop ubuntu --vm --console=vga

The guest is pre-configured to work correctly with SPICE, so that means clipboard integration, automatic resizing with the viewer window, USB redirection, etc. The same also works for other distros, as before:

1
2
3
4
5
6
7
8


# Launch an Arch desktop VM
lxc launch images:archlinux/desktop-gnome arch --vm \
 -c limits.cpu=8 \
 -c limits.memory=16GiB \
 -c security.secureboot=false

# Get a console using a separate command (if preferred!)
lxc console --type=vga arch

LXD UI 😍 #

Back in June 2023, Canonical announced early access to the LXD graphical user interface on their blog. The LXD UI is now generally available and enabled by default from LXD 5.21 onwards - though you can find instructions for enabling it on earlier versions in the docs. The summary is:

1
2
3
4


lxc config set core.https_address :8443

sudo snap set lxd ui.enable=true
sudo systemctl reload snap.lxd.daemon

In my opinion, the LXD UI is one of the best, if not the best way to interact with a hypervisor yet. Being a full-stack web application, it gains independence from different GUI toolkits on Linux and, provided the cluster is remote, can be accessed the same way from Windows, Mac and Linux.

I’ve used other hypervisors with web UIs, particularly Proxmox, and I’ve found the experience with LXD UI to be very smooth, even from the early days. The UI can walk you through the creation and management of VMs, containers, storage and networking. The UI can also give you a nice concise summary of each instance (below is the summary of the VM created using the command in the last section):

One of my favourite features is the web-based SPICE console for desktop VMs, which combined with the management features makes it trivial to stand up a desktop VM and start testing:

Why both? #

By now you’ve probably realised that LXD can do everything Multipass can do, and give much more flexibility - and that’s true. LXD is a full-featured hypervisor which supports much more sophisticated networking, PCI-passthrough, clustering, integration with enterprise identity providers, observability through Prometheus metrics and Loki log-forwarding, etc.

Multipass is small, lean and very easy to configure. If I just want a quick command-line only Ubuntu VM to play with, I still find multipass shell to be most convenient - especially with the automatic home directory mounting.

When I want to work with desktop VMs, interact with non-Ubuntu distributions, or work more closely with hardware, then I use LXD. I was already a bit of a closet LXD fan, having previously described it as a bit of a “secret weapon” for Canonical, but since the introduction of the LXD UI, I’m a fully paid up member of the LXD fan club 😉

Summary #

As I mentioned in the opening paragraphs - both LXD and Multipass have become central to a lot of my technical workflows. The reason I packaged Multipass for NixOS, was that I wanted to dive into daily-driving NixOS, but not without Multipass! In my opinion, the LXD UI is one of the most polished experiences for managing containers and VMs on Linux, and I’m really excited for what that team cooks up next.

Development on Jon Seager

Brewlog: Coffee & Agents

Introduction #

Core Features #

LLM-Powered Bag Scanning #

Design Decisions #

Backend #

Frontend #

Single User, Passkeys Only #

Comprehensive CLI Client #

Domain-Driven Design #

Agentic Coding Patterns #

Pre-commit hooks as a contract #

Stricter lints for agents #

Self-updating instructions #

Plan Mode #

Worktrees and Parallel Agents #

Visual prompting #

Summary #

Supercharging Ubuntu Releases: Monthly Snapshots & Automation

Introduction #

Monthly Snapshot Releases #

Embracing Automation #

Improving Test Coverage #

What’s Next? #

Adopting sudo-rs By Default in Ubuntu 25.10

Introduction #

What is sudo-rs? #

Sponsoring Mainstream Adoption #

Progress on coreutils #

Migration Mechanics #

Summary #

Revitalising Ubuntu Project Documentation

Introduction #

The Challenge #

End Goal #

The Plan #

Summary #

Carefully But Purposefully Oxidising Ubuntu

Introduction #

But… why? #

Introducing oxidizr #

How does it work? #

Get started #

How to Help #

Next Steps #

Summary #

Engineering Ubuntu For The Next 20 Years

Introduction #

Four Key Themes #

Communication #

Automation #

Process #

Modernisation #

Delivering Features #

Summary #

Packaging the Multipass Flutter GUI for NixOS

Introduction #

Housekeeping #

Restructuring #

Using symlinkJoin #

Building the GUI #

Update Script #

Summary #

Experimenting with Rust, Nix, K6 and Parca

Introduction #

Learning Rust #

Rewriting gosherve in Rust #

Basic Load Testing with k6 #

Nix-ified Load Testing #

Defining Test VMs #

Wrapping k6 #

Automating Test Execution #

Initial Load Test Results #

Profiling gosherve with Parca #

Instrumenting gosherve #

Initial optimisations #

Reducing Allocations #

Profiling servy? #

Final Results & Comparisons #

What is `sudo-rs`? #

Progress on `coreutils` #

Introducing `oxidizr` #

Using `symlinkJoin` #

Rewriting `gosherve` in Rust #

Basic Load Testing with `k6` #

Wrapping `k6` #

Profiling `gosherve` with Parca #

Instrumenting `gosherve` #

Profiling `servy`? #

`manifest.json` #

`init.py` #

`config_flow.py` #

`coordinator.py` #

`climate.py` #

`uv` #

`pydantic` #

`ruff` #

Contributing to `nixpkgs` #