Linux on Jon Seager

ntpd-rs: it's about time!

Thu, 26 Mar 2026 00:00:00 +0000

Introduction #

I am thrilled to announce the next target in our campaign to replace core system utilities with memory-safe Rust rewrites in Ubuntu. In upcoming releases, Ubuntu will be adopting ntpd-rs as the default time synchronization client and server, eventually replacing chrony, linuxptp and with any luck, gpsd for time syncing use-cases.

ntpd-rs is a full-featured implementation of the Network Time Protocol (NTP), written entirely in Rust. Maintained by the Trifecta Tech Foundation as part of Project Pendulum, ntpd-rs places a strong focus on security, stability, and memory safety.

To deliver on this goal, we’re building on our partnership with the Trifecta Tech Foundation who are behind sudo-rs, zlib-rs and more. We will be funding the Trifecta Tech Foundation to build new features, enhance security isolation, and ultimately deliver a unified, memory-safe time synchronization utility for the Linux ecosystem. This work meshes well with the Trifecta Tech Foundations goals to improve the security of time synchronization everywhere.

NTP, NTS, and PTP #

Before diving into the mechanics and reasoning behind the transition, I wanted to give some background on the protocols at play, and the problems we’re hoping to solve. Keeping accurate time is a critical system function, not least because it involves constant interaction with the internet and forms the basis for cryptographic verification in protocols such as Transport Layer Security (TLS).

NTP (Network Time Protocol) is the foundational protocol that most operating systems implement to accurately determine the current time from a network source.

NTS (Network Time Security) is to NTP what HTTPS is to HTTP. Historically, the Network Time Protocol was used unencrypted, like many of the early web protocols. NTS introduces cryptographic security to time synchronization, ensuring that bad actors cannot intercept or spoof time data. We already pushed to make NTS the default out-of-the-box in Ubuntu 25.10, which we accomplished by migrating away from ntpd to chrony as the default time-syncing implementation.

PTP (Precision Time Protocol) is used for systems that require sub-microsecond synchronization. While the precision offered by a standard NTP deployment is sufficient for general-purpose computing, PTP is often used for complex, specialized deployments like telecommunications networks, power grids, and automotive applications.

Proven at Scale #

Transitioning core utilities in Ubuntu comes with a responsibility to ensure that replacements are of high quality, resilient and offer something to the platform. We may be the first major Linux distribution to adopt ntpd-rs by default, but we aren’t the first to recognize the readiness of ntpd-rs - it has already been proven at scale by Let’s Encrypt.

While Let’s Encrypt’s core Certificate Authority software has always been written in memory-safe Go, their server operating systems and network infrastructure historically relied on memory-unsafe languages like C and C++, which routinely led to vulnerabilities requiring patching.

Following extensive development, ntpd-rs was deployed to Let’s Encrypt’s staging environment in April 2024, and rolled out to full production by June 2024, marking a major milestone for ntpd-rs.

The fact that one of the world’s most prolific and security-conscious certificate authorities trusts ntpd-rs to keep time across its fleet should provide us, and our enterprise customers, with tremendous confidence in its resilience and suitability.

A Single, Memory-Safe Utility for NTP and PTP #

We want to provide a single utility for configuring both NTP/NTS and Precision Time Protocol (PTP) on Linux. The Trifecta Tech Foundation is concurrently developing Statime, a memory-safe PTP implementation that delivers synchronization performance on par with linuxptp, but with the goal of being easier to configure and use.

The goal is to integrate Statime’s PTP capabilities directly into ntpd-rs, improving the user experience by bringing all time synchronization concerns into one utility with common configuration and usage patterns, obviating the need for complex manual configuration (and troubleshooting) that users of linuxptp may be familiar with.

Timelines and Goals #

As with our transition to sudo-rs and uutils coreutils, leading the mainstream adoption of foundational system utilities comes with responsibility. We want to ensure that ntpd-rs matches the security isolation and performance standards our users expect from chrony.

Canonical is funding the Trifecta Tech Foundation’s development efforts toward these goals over the coming cycles. This work will take place between July 2026 and January 2027 in several major milestones. Our current timeline and targeted goals are:

Ubuntu 26.10: If all goes well, we aim to land the latest version of ntpd-rs in the archive, making it available to test.
Ubuntu 27.04: By 27.04, ntpd-rs should have integrated statime, and we will ship the unified client/server binary for NTP, NTS and PTP in Ubuntu by default, with the aim of providing a smooth migration path for those who already manage complex chrony configs.

To get us there, the Trifecta Tech Foundation will be working on the following items:

Feature Parity & Hardware Support: Adding gpsd IP socket support, multi-threading support for NTP servers, and support for multi-homed servers.
Security & Isolation: chrony is isolated via AppArmor and seccomp. We’ll be working on robust AppArmor and seccomp profiles for ntpd-rs to ensure we don’t buy memory safety at the cost of system-level privilege boundaries. We are also ensuring rustls can use openssl as a crypto provider to satisfy strict corporate cryptography policies.
PTP & Automotive Profiles: Adding support for gPTP, which will allow us to support complex deployments like the Automotive profile directly from nptd-rs (via Statime). Additionally, experimental support for the proposed Client-Server PTP protocol (CSPTP, IEEE 1588.1) will be added.
Benchmarking & Testing: Comprehensive benchmarking of long-term memory, CPU usage, and synchronization performance against chrony to give our cloud partners and enterprise users complete confidence in the transition.
User-experience: Logging improvements and enhancements to configuration that help users configure the time synchronisation target to optimise network usage, as well as improvements to the ntp-cli

About the Trifecta Tech Foundation #

Trifecta Tech Foundation is a non-profit and a Public Benefit Organisation (501(c)(3) equivalent) that creates open-source building blocks for critical infrastructure software. Their initiatives on data compression, time synchronization, and privilege boundary, impact the digital security of millions of people. If you’d like to support their work, please contact them via https://trifectatech.org/support.

Summary #

I am really excited to deepen our already productive relationship with the Trifecta Tech Foundation to make these transitions viable for the wider ecosystem. We’ll be working hard on testing and integration to ensure seamless migration paths, and heavily document the changes ahead of the 26.10 and 27.04 releases.

Stay tuned!

An update on upki

Mon, 16 Feb 2026 00:00:00 +0000

Last year, I announced that Canonical had begun supporting the development of upki, a project that will bring browser-grade Public Key Infrastructure (PKI) to Linux. Since then, development has been moving at pace thanks to the tireless work of Dirkjan and Joe.

In this post, I’ll explore the progress we’ve made, how you can try an early version, and where we’re going next.

Architecture & Progress #

As a reminder, upki’s primary goal is to provide a reliable, privacy-preserving, and efficient certificate revocation mechanism for Linux system utilities, package managers, and language runtimes. The solution is built around CRLite, an efficient data format that compresses and distributes certificate revocation information at scale.

The upki repository is structured as a Cargo workspace containing five crates, each serving a distinct role:

upki: the core library and CLI tool. This crate contains the revocation query engine, the client-side sync logic for fetching filter updates, and the command-line interface. The revocation interface was originally embedded in the CLI, but has since been promoted into the library so that other Rust projects can use it directly as a dependency.
upki-mirror: the server-side mirroring tool. This binary fetches and validates CRLite filters from Mozilla’s infrastructure such that they can be served using a standard web server like nginx or apache.
upki-ffi: the C Foreign Function Interface. Built as a cdylib, this crate uses cbindgen to auto-generate a upki.h header file, exposing the revocation query API to C, C++, Go and any other language with C FFI support.
rustls-upki: an integration crate that wires upki’s revocation engine into rustls, enabling any Rust application using rustls to perform CRLite-backed revocation checks transparently.
revoke-test: testing infrastructure for validating revocation queries against known-revoked certificates.

The team recently released v0.1.0, which should help us to gather more feedback on the work we’ve done so far.

How to try it #

If you’d like to try the code in its current form, you’ll need to have a version of the Rust toolchain installed. The easiest way to do this on Ubuntu is using the rustup snap:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# Ensure you have a C compiler in your PATH
sudo apt update
sudo apt install -y build-essential curl

# Install the rustup snap and get the stable toolchain
sudo snap install --classic rustup
rustup install stable

# Install upki
cargo install upki
export PATH="$HOME/.cargo/bin:$PATH"

# Fetch revocation data. This will be done in the background
# when installed through the distro in the future
upki fetch

That should be all you need to install the development version of upki, and you can now use it to run a revocation check by piping certificate output from curl into upki:

1
2


curl -sw '%{certs}' https://google.com | upki revocation check
NotRevoked

Early versions of docs for the C FFI crate and Rust crate documentation are available, but if you’d like to explore, build the project from source, or contribute, the repository is the best place to start. For an example of the C FFI interface in action you can take a look at the upki-go-demo Dirkjan published.

Next Steps #

Now the foundational pieces are in place, our focus is shifting to external consumption, performance, and integration with the wider Linux ecosystem. In the coming days there should be an early 0.1.0 binary release.

We’ll also be doing some performance benchmarking on the initial fetch and of the revocation checks themselves. Currently, each revocation check reads several CRLite filter files into memory. There may be quick wins to improve this, but we’ll benchmark first and see if it warrants optimisation at this time.

We also need to deploy some production infrastructure for serving the CRLite filters. If you follow the steps above, you’ll be fetching from a pre-production web server hosted at https://upki.rustls.dev. We’ve built a Juju charm for operating the CRLite mirror on Kubernetes. This charm packages the upki-mirror binary in a chiselled Rock, and will be deployed into Canonical’s datacentres to serve CRLite data at crlite.ubuntu.com.

Our Ubuntu Foundations team is also working on packaging the various upki components for inclusion in the Ubuntu archive, which will enable you to apt install upki in the future, and also enable us to package and enable it by default in Ubuntu 26.10 and beyond.

Further Down the Road #

While the work above covers what’s immediately in front of us, there is scope to expand upki’s capabilities further. Two areas of interest are Certificate Transparency enforcement, and support for Merkle Tree Certificates.

Certificate Transparency Enforcement #

While upki’s initial focus is on revocation checking, the project also aims to eventually support Certificate Transparency (CT) enforcement. CT is a more modern security measure that relies upon a set of publicly auditable, append-only logs that record every TLS certificate issued by a Certificate Authority (CA). This prevents CAs from issuing fraudulent or erroneous certificates without a means for that fraudulent activity to be discovered - a problem that has bitten organisations in the past.

CT Enforcement would enable clients to refuse to establish a connection unless the server provides cryptographic proof that its certificate has been correctly logged. Browsers like Chrome and Firefox already enforce this, but the rest of the Linux ecosystem would need a tool such as upki to enable such functionality.

Intermediate Preloading #

A correctly configured TLS server should not only send its own certificate, but also the intermediate certificates needed to chain back to a trusted root. In practice, many servers omit the intermediate certificates, and because browsers have quietly worked around this for years, the misconfiguration often goes unnoticed.

Firefox has been preloading all intermediates disclosed to the Common CA Database (CCADB) since Firefox 75, while Chrome and Edge will silently fetch missing intermediates using the Authority Information Access (AIA) extension in the server’s certificate. The result is that a broken certificate chain that works perfectly in every browser will produce an opaque UNKNOWN_ISSUER error when accessed by Linux utilities like curl.

Because upki already maintains a regularly synced local data store, it’s well positioned to ship the known set of intermediates alongside the CRLite filters. This wouldn’t provide a security improvement so much as a usability improvement. It would also bring non-browser clients up to parity with browsers with respect to connection reliability. There is an additional privacy benefit too: rather than fetching a missing intermediate from the issuing CA (which discloses browsing activity to the CA), the intermediate is already present locally.

Merkle Tree Certificates #

Looking even further ahead, upki could support the next generation of web PKI by including support for Merkle Tree Certificates (MTCs). This is an area of active development in the IETF, with Cloudflare and Chrome recently announcing an experimental deployment.

The motivation for MTCs comes largely from the transition to Post-Quantum (PQ) cryptography. PQ signatures are significantly larger than their non-PQ counterparts. The signatures for ML-DSA-44 are 2,420 bytes compared to 64 bytes for ECDSA-P256. A typical TLS handshake today involves multiple signatures and public keys across the certificate chain and CT proofs, which means a simple swap to PQ algorithms would add tens of kilobytes of overhead per connection and likely a noticeable increase in connection latency.

MTCs address this by rethinking how certificates are validated. Rather than transmitting a full certificate chain with multiple signatures, a Certificate Authority can batch certificates into a Merkle Tree and sign only the tree’s root hash. The client then receives just a single signature, a public key, and a compact Merkle tree inclusion proof that demonstrates the certificate’s presence in the batch. The signed tree heads can be distributed to clients out-of-band, meaning the per-handshake overhead is drastically reduced.

Because upki already maintains a local data store that is regularly synced, it could cache tree head data alongside CRLite filters, thereby enabling the inclusion proofs sent during TLS handshakes to be even smaller. Rather than proving inclusion all the way from the leaf to the root, the server could send a “truncated” proof that starts partway up the tree, with the client computing the remainder from data it already has locally. There is a TLS extension being developed to negotiate this.

The implementation of MTCs for TLS is still highly experimental. MTCs are not yet deployed in any browser, but upki will lay the groundwork for Linux system utilities to benefit from this evolution as the technology is adopted.

Summary #

In the few weeks since we announced upki, the core revocation engine has been established and is now functional, the CRLite mirroring tool is working and a production deployment in Canonical’s datacentres is ongoing. We’re now preparing for an alpha release and remain on track for an opt-in preview for Ubuntu 26.04 LTS.

Beyond revocation, we’re keeping a close eye on the evolving PKI landscape and particularly CT enforcement and Merkle Tree Certificates.

I’d like to extend my thanks again to Dirkjan and Joe for their continued collaboration on this work, and the utmost professionalism they’ve demonstrated throughout.

Developing with AI on Ubuntu

Tue, 20 Jan 2026 00:00:00 +0000

AI-assisted tooling is becoming more and more common in the workflows of engineers at all experience levels. As I see it, our challenge is one of consideration, enablement and constraint. We must enable those who opt-in to safely and responsibly harness the power of these tools, while respecting those who do not wish to have their platform defined or overwhelmed by this class of software.

The use of AI is a divisive topic among the tech community. I find myself a little in both camps, somewhere between sceptic and advocate. While I’m quick to acknowledge the negative impacts that the use of LLMs can have on open source projects, I’m also surrounded by examples where it has been used responsibly to great effect.

Examples of this include Filippo’s article debugging low-level cryptography with Claude Code, Mitchell’s article on Vibing a Non-Trivial Ghostty Feature, and David’s article How I Program with Agents. These articles come from engineers with proven expertise in careful, precise software engineering, yet they share an important sentiment: AI-assisted tools can be a remarkable force-multiplier when used in conjunction with their lived experience, but care must still be taken to avoid poor outcomes.

The aim of this post is not to convince you to use AI in your work, but rather to introduce the elements of Ubuntu that make it a first-class platform for safe, efficient experimentation and development. My goals for AI and Ubuntu are currently focused on enabling those who want to develop responsibly with AI tools, without negatively impacting the experience of those who’d prefer not to opt-in.

Hardware & Drivers #

AI-specific silicon is moving just as fast as AI software tooling, and without constant work to integrate drivers and userspace tools into Ubuntu, it would be impossible to efficiently utilise this specialised hardware.

Last year we announced that we will ship both NVIDIA’s CUDA and AMD’s ROCm in the Ubuntu archive for Ubuntu 26.04 LTS, in addition to our previous work on OpenVINO. This will make installing the latest drivers and toolkits easier and more secure, with no third-party software repositories. Distributing this software as part of Ubuntu enables us to be proactive in the delivery of security updates and the demonstration of provenance.

Our work is not limited to AMD and NVIDIA; we recently announced support for Qualcomm’s Dragonwing platforms and others. You can read more about our silicon partner projects on our website.

Inference Snaps #

At the Ubuntu Summit 25.10, we released “Inference Snaps” into the wild, which provide a hassle-free mechanism for obtaining the “famous model” you want to work with, but automatically receive a version of that model which is optimised for the silicon in your machine. This removes the need to spend hours on HuggingFace identifying the correct model to download that matches with your hardware, and obviates the need for in-depth understanding of model quantisation and tuning when getting started.

Each of our inference snaps provide a consistent experience: you need only learn the basics once, but can apply those skills to different models as they emerge, whether you’re on a laptop or a server.

At the time of writing, we’ve published beta quality snaps for qwen-vl, deepseek-r1 and gemma3. You can find a current list of snaps in the documentation, along with the silicon-optimised variants.

Sandboxing Agents #

While many start their journey in a web browser chatting to ChatGPT, Claude, Gemini, Perplexity or one of the myriad of alternatives, many developers will find “agentic” tools such as Copilot, Codex, Claude Code or Amp quite attractive. In my experience, agents are a clear level-up in an LLM’s capability for developers, but they can still make poor decisions and are generally safer to run in sandboxed environment at the time of writing.

Where a traditional chat-based AI tool responds reactively to user prompts within a single conversation, an agent operates (semi-)autonomously to pursue goals. It perceives its environment, plans, makes decisions and can call out to external tools and services to achieve those goals. If you grant permission, an agent can read and understand your code, implement features, troubleshoot bugs, optimise performance and many other tasks. The catch is that they often need access to your system - whether that be to modify files or run commands.

Issues such as accidental file deletion, or the inclusion of a spurious (and potentially compromised) dependency are an inevitable failure mode of the current generation of agents due to how they’re trained (see the Reddit post about Claude Code deleting a user’s home directory).

My agent sandboxes itself! #

Some of you will be reading this wondering why additional sandboxing is required, since many of the popular agents advertise their own sandboxing. The fact that some agents include some measures to protect the user’s machine is of course a good thing. The touted benefits include filesystem isolation by restricting the agent to a specific directory, or prompting for approval before modifying files. Some agents also include network sandboxing to restrict network access to a list of approved domains, or by using a custom proxy to impose rules on outbound traffic.

On Linux, these agent-imposed sandboxes are often implemented with bubblewrap, which is “a tool for constructing sandbox environments”, but note that the upstream project’s README includes a section which states that it is not a “ready-made sandbox with a specific security policy”. bubblewrap is a relatively low-level tool that must be given its configuration, which in this case is provided by the agent.

The limitation upon these tools is the shared kernel - a severe kernel exploit could enable an agent to escape from its sandbox. Of course, such vulnerabilities are rare, but note that even if the sandboxing technologies do their job, agents often run in the context of the user’s session, meaning they inherit environment variables which could contain sensitive information. They’re also agent specific: Claude Code’s sandboxing won’t help you if you’re using Cursor or Antigravity.

Depending on your threat model and the project you’re working on, you may deem the built-in sandboxing of coding agents to be sufficient, but there are other options available to Ubuntu users that provide either different, or additional protection…

Sandbox with LXD containers #

Canonical’s LXD works out-of-the-box on Ubuntu, and is a great way to sandbox an agent into a disposable environment where the blast radius is limited should the agent make a mistake. My personal workflow is to create an Ubuntu container (or VM) with my project directory mounted. This way, I can edit my code directly on my filesystem with my preferred (already configured) editor, but have the agent run inside the container.

For example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Initialise the container
lxc init ubuntu:noble dev
# Mount my project directory into the container
lxc config device add -q dev datadir disk source="$HOME/my-project" path=/home/ubuntu/project
# Start the container
lxc start dev
# Get a shell inside the container as the 'ubuntu' user
lxc exec dev -- sudo -u ubuntu -i bash
# Run a command in the container
lxc exec dev -- sudo -u ubuntu -i bash -c "cd project; claude"

You can learn more about LXD in the official documentation and tutorial, as well as specific instructions on enabling GPU data processing in containers/VMs. I’ve written previously about my use of LXD in development.

With LXD, you can choose between running your sandbox as a container or a VM, depending on your project’s needs. If I’m working on a project that requires Kubernetes or similar, I use a VM, but for lighter projects I use system containers, preferring their lower overhead.

Sandbox with LXD VMs #

LXD is best known for its ability to run “system containers”, which are somewhat analogous to Docker/OCI containers, but rather than being focused on a single application (and dependencies), a system container essentially runs an entire Ubuntu user-space (including systemd, etc.). Like OCI containers, however, system containers share the kernel with the host.

In some situations, you may seek more isolation from your host machine by running tools inside a virtual machine with their own kernel. LXD makes this simple - you can follow the same commands as above, but add --vm to the init command:

1
2


# Initialise the virtual machine
lxc init --vm ubuntu:noble dev

You can also configure the virtual machine’s CPU, memory and disk requirements. A simple example is below:

1
2
3
4


lxc init --vm ubuntu:noble dev \
 -c limits.cpu=8 \
 -c limits.memory=8GiB \
 -d root,size=100GiB

You can find more details on instance configuration in the LXD documentation.

Sandbox with Multipass #

Multipass provides on-demand access to Ubuntu VMs from any workstation - whether that workstation is running Linux, macOS or Windows. It is designed to replicate, in a lightweight way, the experience of provisioning a simple Ubuntu VM on a cloud.

Multipass’ scope is more limited than LXD, but for many users it provides a simple on-ramp for development with Ubuntu. Where it lacks advanced features like GPU passthrough, it boasts a simplified CLI and a first-class GUI client.

To get started similarly to the LXD example above, try the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Install Multipass
sudo snap install multipass
# Launch an instance
multipass launch noble -n dev
# Mount your project directory
multipass mount ~/my-project dev:/home/ubuntu/project
# Get a shell in the instance
multipass shell dev
# Run a command in the instance
multipass exec dev -- claude

You can find more details on how to configure and manage instances in the docs.

Sandbox with WSL #

If you’re on Windows, development with WSL includes first-class support for GPU acceleration, and is even supported for use with the NVIDIA AI Workbench, NVIDIA NIM and CUDA.

Ubuntu is the default Linux distribution for WSL, and you can find more information about how to set up and use Ubuntu on WSL in our documentation. WSL benefits from all the same technologies as a “regular” Ubuntu install, including the ability to use Snaps, Docker and LXD.

For the enterprise developer, we recently announced Ubuntu Pro for WSL, as well as the ability to manage WSL instances using Landscape, making it easier to get access to first-class developer tooling with Ubuntu on your corporate machine.

Summary #

While opinion remains divided on the value and impact of current AI tooling, its presence in modern development workflows and its demands on underlying compute infrastructure are difficult to ignore.

Developers who wish to experiment need reliable access to modern hardware, predictable tooling, and strong isolation boundaries. Ubuntu’s role is not to dictate how these tools are used, but to provide a stable and dependable platform on which they can be explored and deployed safely, without compromising security, provenance, or the day-to-day experience of those who choose to opt out.

In addition to powering development workflows, Ubuntu makes for a dependable production operating system for your workloads. We’re building Canonical Kubernetes with first-class GPU support, Kubeflow and MLFlow for model training and serving and a suite of applications like PostgreSQL, MySQL, Opensearch, as well as other data-centric tools such as Kafka and Spark that can be deployed with full Ubuntu Pro support. Let me know if you’d find value in a follow-up post on those topics!

Addressing Linux's Missing PKI Infrastructure

Mon, 08 Dec 2025 00:00:00 +0000

Earlier this year, LWN featured an excellent article titled “Linux’s missing CRL infrastructure”. The article highlighted a number of key issues surrounding traditional Public Key Infrastructure (PKI), but critically noted how even the available measures are effectively ignored by the majority of system-level software on Linux.

One of the motivators for the discussion is that the Online Certificate Status Protocol (OCSP) will cease to be supported by Let’s Encrypt. The remaining alternative is to use Certificate Revocation Lists (CRLs), yet there is little or no support for managing (or even querying) these lists in most Linux system utilities.

To solve this, I’m happy to share that in partnership with rustls maintainers Dirkjan Ochtman and Joe Birr-Pixton, we’re starting the development of upki: a universal PKI tool. This project initially aims to close the revocation gap through the combination of a new system utility and eventual library support for common TLS/SSL libraries such as OpenSSL, GnuTLS and rustls.

The Problem #

Online Certificate Authorities responsible for issuing TLS certificates have long had mechanisms for revoking known bad certificates. What constitutes a known bad certificate varies, but generally it means a certificate was issued either in error, or by a malicious actor of some form. There have been two primary mechanisms for this revocation: Certificate Revocation Lists (CRLs) and the Online Certificate Status Protocol (OCSP).

In July 2024, Let’s Encrypt announced the deprecation of support for the Online Certificate Status Protocol (OCSP). This wasn’t entirely unexpected - the protocol has suffered from privacy defects which leak the browsing habits of users to Certificate Authorities. Various implementations have also suffered reliability issues that forced most implementers to adopt “soft-fail” policies, rendering the checks largely ineffective.

The deprecation of OCSP leaves us with CRLs. Both Windows and macOS rely on operating system components to centralise the fetching and parsing of CRLs, but Linux has traditionally delegated this responsibility to individual applications. This is done most effectively in browsers such as Mozilla Firefox, Google Chrome and Chromium, but this has been achieved with bespoke infrastructure.

However, Linux itself has fallen short by not providing consistent revocation checking infrastructure for the rest of userspace - tools such as curl, system package managers and language runtimes lack a unified mechanism to process this data.

The ideal solution to this problem, which is slowly becoming more prevalent, is to issue short-lived credentials with an expiration of 10 days or less, somewhat removing the need for complicated revocation infrastructure, but reducing certificate lifetimes is happening slowly and requires significant automation.

CRLite #

There are several key challenges with CRLs in practice - the size of the list has grown dramatically as the web has scaled, and one must collate CRLs from all relevant certificate authorities in order to be useful. CRLite was originally proposed by researchers at IEEE S&P and subsequently adopted in Mozilla Firefox. It offers a pragmatic solution to the problem of distributing large CRL datasets to client machines.

In a recent blog post, Mozilla outlined how their CRLite implementation meant that on average users “downloaded 300kB of revocation data per day, a 4MB snapshot every 45 days and a sequence of “delta-updates” in-between”, which amounts to CRLite being 1000x more bandwidth-efficient than daily CRL downloads.

At its core, CRLite is a data structure compressing the full set of web-PKI revocations into a compact, efficiently queryable form. You can find more information about CRLite’s design and implementation on Mozilla’s Security Blog.

Introducing upki #

Following our work on oxidizing Ubuntu, Dirkjan reached out to me with a proposal to introduce a system-level utility backed by CRLite to non-browser users.

upki will be an open source project, initially packaged for Ubuntu but available to all Linux distributions, and likely portable to other Unix-like operating systems. Written in Rust, upki supports three roles:

Server-side mirroring tool: responsible for downloading and mirroring the CRLite filters provided by Mozilla, enabling us to operate independent CDN infrastructure for CRLite users, and serving them to clients. This will insulate upki from changes in the Mozilla backend, and enable standing up an independent data source if required. The server-side tool will manifest as a service that periodically checks the Mozilla Firefox CRLite filters, downloads and validates the files, and serves them.
Client-side sync tool: run regularly by a systemd-timer, network-up events or similar, this tool ensures the contents of the CDN are reflected in the on-disk filter cache. This will be extremely low on bandwidth and CPU usage assuming everything is up to date.
Client-side query tool: a CLI interface for querying revocation data. This will be useful for monitoring and deployment workflows, as well as for users without a good C FFI.

The latter two roles are served by a single Rust binary that runs in different modes depending on how it is invoked. The server-side tool will be a separate binary, since its use will be much less widespread. Under the hood, all of this will be powered by Rust library crates that can be integrated in other projects via crates.io.

For the initial release, Canonical will stand up the backend infrastructure required to mirror and serve the CRLite data for upki users, though the backend will be configurable. This prevents unbounded load on Mozilla’s infrastructure and ensures long-term stability even if Firefox’s internal formats evolve.

Ecosystem Compatibility #

So far we’ve covered the introduction of a new Rust binary (and crate) for supporting the fetching, serving and querying of CRL data, but that doesn’t provide much service to the existing ecosystem of Linux applications and libraries in the problem statement.

The upki project will also provide a shared object library for a stable ABI that allows C and C-FFI programs to make revocation queries, using the contents of the on-disk filter cache.

Once upki is released and available, work can begin on integrating existing crypto libraries such as OpenSSL, GNUtls and rustls. This will be performed through the shared object library by means of an optional callback mechanism these libraries can use to check the revocation lists before establishing a connection to a given server with a certificate.

Timeline #

While we’ve been discussing this project for a couple of months, ironing out the details of funding and design, work will soon begin on the initial implementation of upki.

Our aim is to make upki available as an opt-in preview for the release of Ubuntu 26.04 LTS, meaning we’ll need to complete the implementation of the server/client functionality, and bootstrap the mirroring/serving infrastructure at Canonical before April 2026.

In the following Ubuntu release cycle, the run up to Ubuntu 26.10, we’ll aim to ship the tool by default on Ubuntu systems, and begin work on integration with the likes of NSS, OpenSSL, GNUtls and rustls.

Summary #

Linux has a clear gap in its handling of revocation data for PKIs. Over the coming months we’re hoping to address that gap by developing upki not just for Ubuntu, but for the entire ecosystem. Thanks to Mozilla’s work on CRLite, and the expertise of Dirkjan and Joe, we’re confident that we’ll deliver a resilient and efficient solution that should make a meaningful contribution to systems security across the web.

If you’d like to do more reading on the subject, I’d recommend the following:

LWN.net: Linux’s missing CRL infrastructure
Mozilla Security Blog: CRLite Part 1: All Web PKI Revocations Compressed
Mozilla Security Blog: CRLite Part 2: End-to-End Design
Let’s Encrypt: Replacing OCSP with CRLs
IEEE Symposium on Security & Privacy: CRLite: A Scalable System for Pushing All TLS Revocations to All Browsers

Ubuntu Summit 25.10: Personal Highlights

Sun, 02 Nov 2025 00:00:00 +0000

I recently had the privilege of attending the Ubuntu Summit 25.10. Ubuntu Summits have a relatively long history. Some years ago Canonical ran the ‘Ubuntu Developer Summits (UDS)’, but recently the events were brought back and reimagined as the ‘Ubuntu Summit’.

For the most recent Summit, we tried out a new format. We invited a select few folks to come and give talks at our London office, with a small in-person crowd. In addition, the event was livestreamed, and we encouraged people to host “watching parties” across the world as part of Ubuntu Local Communities (LoCos).

While Ubuntu may feature in the name, the event does not require talks to be centred on Ubuntu, and in fact is aiming to draw contributions from our partners and from right across the open source community, whether or not the content is relevant to Ubuntu or Canonical - it’s designed to be a showcase for the very best of open source, and this year I felt that the talks were of a particularly high calibre.

In this post I’ll highlight some of my favourite talks, in no particular order! If any of these catch your interest, you can see when they were aired and catch-up on the Day 1 and Day 2 streams.

DOOM in Space #

What a way to kick off the Summit! DOOM in Space was a talk given by Ólafur Waage, who introduced himself as a “professional keyboard typist”!

The talk was immediately after Mark Shuttleworth’s opening remarks, and covered his journey in getting DOOM to run on the European Space Agency’s OPS-SAT satellite. DOOM has famously been ported to many devices, though some were only questionably “running” the game.

Ólafur covered how he became involved in the project, and the unique approach they needed to take to guarantee success, since they would only get a very limited amount of time in order to conduct their “experiment” on the satellite.

Of particular note was the work done to integrate imagery from the OPS-SAT’s onboard camera into the game, which involved some clever reassigning of colors in the game’s original palette to more faithfully represent the imagery taken from the camera in-game.

Infrastructure-Wide Profiling of Nvidia CUDA #

This talk was given by Frederic Branczyk, CEO and Founder of Polar Signals. Canonical has partnered with Polar Signals a couple of times in recent years. They were part of our journey to enabling frame pointers by default on Ubuntu, and many of our teams have been using their zero-instrumentation eBPF profiler.

While CPU profiling has been commonplace for developers for many years, giving the ability to analyse CPU and memory-bound workloads, profiling GPU workloads has been less prominent, and is particularly difficult in production.

Polar Signals advocate for “continuous profiling”, which means running a profiler at all times, on all nodes, in production. The benefit of this is that when an issue occurs, you don’t have to set up a profiler and try to reproduce the issue - you already have the data. It also negates the uncertainty of the impact a profiler might have on the code during reproduction. This would have been difficult with traditional profiling tools, but with technologies like eBPF, the overhead of the profiler is incredibly low compared to the potential performance gains from acting on the data it produces.

In this talk, Frederic outlined the work they have done bringing infrastructure-wide profiling of CUDA workloads into Polar Signals Cloud. Their approach combines the CUPTI profiling API with USDT probes and eBPF into a pipeline, relying upon the ability to inject a small library into CUDA workloads using the CUDA_INJECTION64_PATH without modification.

You can see more details on their website.

Inference Snaps #

This talk served as the first public announcement of Inference Snaps from Canonical, which represents a few months of working combining many of the new technologies behind Snaps.

As Large Language Models continue to gain pace along with the rest of the AI community, silicon manufacturers are increasingly including dedicated hardware in commodity CPUs and GPUs, as well as shipping dedicated accelerators for some workloads.

AI models often need to be tuned in some way in order to work optimally - for example quantisation which aims to reduce the computational memory costs of running inference on a given model.

Inference snaps provide a hassle-free mechanism for users to obtain the “famous model” they want to work with, but automatically receive a version of that model which is optimised for the silicon in their machine, removing the need to spend hours on HuggingFace trying to identify the correct model to download that matches with their hardware.

Using our extensive partner network, we’ll continue to work with multiple silicon vendors to ensure that models are available for the latest hardware as it drops, and provide a consistent experience to Ubuntu users that wish to work with AI.

Find out more in the announcement.

Nøughty Linux: Ubuntu’s Stability Meets Nixpkgs’ Freshness #

This talk was a bit of a guilty pleasure for me! Delivered by Martin Wimpress (wimpy), the audience were shown how they could take a stock Ubuntu Server deployment, and use a collection of scripts to layer a cutting-edge GUI stack on top using Nixpkgs.

Wimpy outlined his motivation as wanting to rely upon the stable kernel and hardware support offered by Ubuntu, but wanting to be more experimental with his desktop environment and utilities - preferring a tiling window management experience.

Having spent some years on NixOS, Wimpy was recently required to run a security “agent” for work, which was very difficult to enable on NixOS, but worked out of the box on Ubuntu. Recognising the need to make the switch, he was reluctant to move away from the workflow he’d built so much muscle-memory around - and so Nøughty Linux was born!

Nøughty Linux is not a Linux distribution, rather a set of configurations for an Ubuntu Server machine. It utilises nix-system-graphics and system-manager and is actually very similar to a configuration I ran in my own nixos-config repository for my laptop for a while - though Wimpy has chased down significantly more of the papercuts than I did!

Are we stuck with the same Desktop UX forever? #

Scott Jenson delivered an incredibly engaging talk in which he posited that desktop user experience has somewhat stagnated, and worse that many of the patterns we’ve become used to on the desktop are antiquated and unergonomic.

The crux of the talk was to focus on user experience, rather than user interfaces - challenging developers to think about how people learn, and how desktops could benefit more from design affordances by rethinking some critical elements such as window management or text editing.

Using his years of experience at Apple, Symbian and Google, Scott delivered one of the most engaging conference talks I’ve seen, and I thoroughly recommend watching it on our YouTube channel!

Honorable Mentions #

In addition to the talks above, it was a delight to meet Mark Schoolderman from the Trifecta Tech Foundation in-person, who led the work on sudo-rs as part of our “Oxidising Ubuntu” story, and interesting to hear about the value the project derived from Ubuntu’s Main Inclusion Review process as part of landing sudo-rs in main.

Equally, I was delighted that Samuele Kaplun from Proton could join us to talk about the work we’ve been doing together on bringing first-class Snap packages for Proton Mail, Proton VPN, Proton Pass and Proton Authenticator to the Snap store, and their reasons for choosing Snaps, adventures with confinement, and more.

I was delighted to see Craig Loewen and Clint Rutkas present on their journey open sourcing the Windows Subsystem For Linux (WSL), which represents a growing proportion of Ubuntu users, and a key bridge to open source development for many.

Finally, thank you to Utkarsh for this wonderful slide as part of his talk on Ubuntu Snapshot Releases:

Conclusion #

Overall, I found the Ubuntu Summit 25.10 a really enjoyable event, with talks that were uniformly high in quality, charisma and creativity. I’m pleased that Canonical has broadened the Summit’s reach and I hope it continues to serve as a platform to showcase the very best open source innovation.

Until next time!

Ubuntu Engineering in 2025: A Retrospective

Thu, 09 Oct 2025 00:00:00 +0000

Ubuntu 25.10: A Retrospective #

In February this year, I published Engineering Ubuntu For The Next 20 Years, which was something of a manifesto I pledged to enact in the design, build and release of Ubuntu. This week, we released Ubuntu 25.10 Questing Quokka, which was the first full engineering cycle under this new manifesto, and it seems like a good time to reflect on what we achieved in each category, as well as highlight some of the more impactful changes that have just landed in Ubuntu.

In that first article, I outline four themes for Ubuntu Engineering at Canonical to focus on: Communication, Automation, Process and Modernisation.

Communication #

A notable improvement throughout this engineering cycle has been the frequency with which the teams at Canonical have written about their work, often in some detail. Many of these posts can be found under the blog tag, which had never been used until around six months ago, and now sees a couple of new posts per week outlining the work people are doing toward these themes.

I stated that I consider documentation a key part of our communication strategy, and this last six months has seen some of the most substantial changes to Ubuntu documentation in many years. The Ubuntu Project Docs was a project started in May 2025, and is quickly becoming the single documentation hub that a current or potential Ubuntu contributor needs to understand how, why and when to do their job. Similarly, the Ubuntu for Developers was created to illuminate a path for developers across numerous languages on Ubuntu.

It’s important for us to celebrate such efforts, but also to remember that this is only the start! In order for these efforts to remain useful, both our internal teams and our community must continue to engage with these efforts - adding, refining and pruning content as necessary. As the sun-setting of wiki.ubuntu.com approaches, it’s imperative that these new documentation sites continue to get the attention they need.

Lots of the changes we’ve made in the last cycle have attracted attention from online blogs, news outlets, youtubers, etc. Part of the challenge with such changes is “owning the narrative” and ensuring that legitimate concerns are heard (and taken into account), but also that there are appropriate responses to uncertainty, without getting drawn into unproductive discussions.

Finally, the transition to Matrix as the default synchronous communication means for the project has, in my opinion, made it easier than ever to get in touch with our community of experts - whether it be for support, or to start a journey for contribution to Ubuntu.

Automation #

The largest item we took on here was in pursuit of the monthly snapshot releases. This went much better than I expected, and to some extent covers off the “Process” theme as well as “Automation”, but through a combination of studying our process and whittling it down as lean as we could, and beginning to automate more of the process, the team were able to release four snapshot releases before the 25.10 Beta.

The scale of the automation efforts was relatively limited this cycle, but the automation of release testing has really accelerated in the past few months. The vast majority of the test cases that qualify an Ubuntu Desktop ISO for release are now fully automated, and the same framework that makes this possible was also used to develop a suite of tests for TPM FDE.

Work was also done on our craft tools to better the experience with the test sub-command of build tools like snapcraft, rockcraft and charmcraft - all of which will have a trickle-down effect on the upcoming debcraft, and make it trivial to include many new kinds of tests in our packaging workflows.

Behind the scenes, every team in Ubuntu Engineering at Canonical has been writing charms that make the underlying infrastructure behind Ubuntu more portable, resilient and scalable. This includes services like Ubuntu Manpages, autopkgtest, error-tracker, and a staging deployment of Temporal to enable the next phase of our release automation.

Process #

This item was probably where the least concrete progress was made, though I probably could have predicted that. Many of the processes in the Ubuntu project serve to ensure that we ship resilient software, and don’t break users - so changing them in a hurry is not generally a good idea.

That said, there was some good progress on the Main Inclusion Review (MIR) process, whose team documentation was moved into the Ubuntu Project Docs after a thorough review, and the Stable Release Updates (SRU) team are in the process of the same transition. Moving and re-reviewing the documentation is essentially the first step of the process improvement I was seeking: understanding where we are!

Internally, we’ve been piloting a new process for onboarding Ubuntu Developers that sees engineers start by working toward gaining upload rights for a single package, but has a complete curriculum that can take them through to Core Developer status. Details of this should be released in the coming months, outlining a clear and well-trodden journey for new contributors. Much of this material already existed, but the team have worked on polishing it, and making it clearer how the process work from end to end.

The next step for each of these processes is measurement. We’ve begun instrumenting these processes to understand where the most time is spent so we can use that information to guide improvements and streamline processes in future cycles, and even set Service Level Objectives (SLOs) against those timelines.

Modernisation #

Much of what I’ve already described could be considered modernisation, but from a technical standpoint the most obvious candidate here was the “Oxidising Ubuntu” effort, which has seen us replace numerous core utilities in Ubuntu 25.10 with modern Rust rewrites.

We began this effort in close collaboration with the uutils project and the Trifecta Tech Foundation. The former is the maintainer of a Rust coreutils rewrite, and the latter the maintainer of sudo-rs, which we made the default in 25.10. The technical impact of these changes in defaults will only truly be known once Ubuntu 25.10 is “out there”, but I’m pleased with how we approached the shift. In both cases, we contacted the upstreams in good time to ascertain their view on their projects’ readiness, then agreed funding to ensure they had the financial support they needed to land changes in support of Ubuntu, and then worked closely with them throughout the cycle to solve various performance and implementation issues we discovered along the way.

As it stands today, sudo-rs is the default sudo implementation on Ubuntu 25.10, and uutils’ coreutils has mostly replaced the GNU implementation, with a few exceptions, many of which will be resolved by releases in the coming weeks. These diversions back to the existing implementations demonstrate that stability and resilience are more important than “hype” in our approach: I expect us to have completed the migration during the next cycle, but not before the tools are ready.

Following the “Switch to Dracut” specification, Ubuntu Desktop 25.10 will use Dracut as its default initrd infrastructure (replacing initramfs-tools). Dracut will use systemd in the initrd and supports new features like Bluetooth and NVMe over Fabric (NVM-oF) support. Ubuntu Server installations will continue using initramfs-tools until remaining hooks are ported.

For each of these changes (coreutils, sudo-rs and dracut) the previous implementations will remain supported for now, with well-documented instructions on the reversion of each change for those who run into unavoidable issues - though we expect this to be a very small number of cases.

What’s Next? #

Well… more of the same! We intend to carry on with the increased cadence of written updates, so keep an eye out for those.

We have some exciting announcements to make over the coming weeks, including support for more modern micro-architectural variants (like amd64v3), better system-wide handling of revoked TLS certificates, updates on our Debcraft package for a more modern packaging experience and an effort to update many of our tools from “behind the scenes” using a combination of Rust and Go.

My final words are to thank all of those who have driven these efforts. I’ll omit the long list of names, but there have been countless examples of people stepping up substantially to deliver these efforts - without whom we’d have made a lot less progress.

Well done, and let’s make Resolute Raccoon an LTS to remember - for all the right reasons!

The Immutable Linux Paradox

Mon, 01 Sep 2025 00:00:00 +0000

Immutable Linux distributions have been around since the early 2000s, but adoption has significantly accelerated in the last five years. Mainstream operating systems (OSes) such as macOS, Android, ChromeOS and iOS have all embraced similar principles, reflecting a growing trend toward resilience, longevity, and maintainability as core ideals of OS development.

Ubuntu Core has been at the forefront of this movement for IoT, appliances and edge deployments, with work ongoing to release a “Core Desktop” experience. Other projects such as NixOS, Fedora Silverblue and Red Hat image mode are gaining adoption, alongside more specialised immutable distributions such as SteamOS and Talos.

This post explores how different Linux distributions achieve immutability, the trade-offs, and why you should give it a try!

What is an immutable Linux distribution? #

The key principle of an immutable OS is that the core system is unchangeable at runtime.

Every OS installation has at least one filesystem that stores system software, user software, and user data. Immutable OSes must cleanly separate “system” and “user” software and data, such that regular user interactions cannot compromise the integrity of the OS.

Immutable deployments are often separated into three layers:

Base OS - immutable core, updated only through controlled mechanisms
Applications - user applications, often delivered in containerised formats such as Snap, Flatpak, AppImage, cpak
User data - writable and persistent, independent of OS updates or rollbacks

Immutable systems use atomic, transactional updates meaning updates are applied as unitary, indivisible operations that either wholly succeed, or fail completely and trigger an automated roll-back to a previous known-good state.

Why immutability? #

The major benefit of an immutable OS is resilience.

Immutable OSes make it easier to reproduce systems with a given configuration, which is particularly useful in scale-out use-cases such as cloud or IoT.

Traditional package managers often maintain a database of installed packages, consisting of those included in the base OS, and those explicitly installed by the user, and their dependencies. The package manager doesn’t have a clear notion of which packages make up the “core system”, and which are “optional”.

This can cause “configuration drift”, which occurs over time - a package could be explicitly installed by a user, used for a while and then removed, but without removing its dependencies. This leaves the system in a different, and somewhat undefined, state than it was in prior to the package being installed.

Often the traditional notion of OS security is improved with immutable OS concepts too. In most implementations, the core OS files are mounted read-only such that users cannot make changes - which also raises the bar for malicious modifications. When combined with technologies such as secure boot and confinement, immutable OSes can dramatically reduce the attack surface of a machine.

Finally, convenience! Immutable OSes often include recovery or rollback features, which enable users to “undo” a bad system change, reverting to a previous known-good revision.

The immutability paradox #

In reality, no general-purpose operating system is fully immutable.

There is always persistent, user-writable storage - because without this there would be a huge limitation on usefulness! Similarly, how can a system be truly immutable, yet still support software updates?

The terms “immutable” and “stateless” are often conflated - when in reality neither are excellent terms for describing what has become widely known as “immutable OSes”. This was explored in some depth in this blog post which proposes terms such as “image based” and “fully managed”.

By definition, changes to configuration, the installation of applications and the use of temporary runtime storage are all violations of immutability, and thus immutability concepts must be applied in some sort of layering system.

Striking the balance between ’true’ immutability and user experience is one of the hardest challenges in immutable OS design. A system that is too rigid can be difficult to manage and use, appearing inflexible to end users.

A common pattern is to run an immutable desktop OS and use virtualisation or containerisation technologies (e.g. LXD, Podman, toolbx Distrobox) to create mutable environments in which to work on projects. This results in a very stable workstation that benefits from immutability, with the flexibility of a traditional mutable OS where it’s needed.

Approaches to immutability #

Different distributions solve the immutability challenge in different ways. In this section we’ll explore the four different approaches of ostree based distributions, bootc based distributions, NixOS and Ubuntu Core.

Fedora Silverblue / CoreOS / EndlessOS (`ostree`) #

Fedora Silverblue and Fedora CoreOS are also popular choices for those exploring immutable OSes. The two share a lot of underlying technology with Silverblue targeting desktop use cases, and CoreOS targeting server deployments.

Both are based on ostree, which provides:

Silverblue and CoreOS actually rely on rpm-ostree , a “hybrid image/package manager” which combines RPM packaging technology with ostree to manage deployments.

The update mechanism involves switching the filesystem to track a different remote “ref”, which is analogous to a git ref.

EndlessOS is based on Debian, but uses ostree to achieve immutability. EndlessOS is a desktop experience designed more for the “average user” and focuses on providing a reliable system that works well in low-bandwidth or offline situations.

Users often use Flatpak to install graphical user applications atop the immutable base, or a user-space package manager such as brew for other utilities.

ostree based distributions also support “package layering” which enables adding packages to the base system without fetching a whole new filesystem ref, but does require the system to be rebooted before the package is persistently available. The documentation notes that this approach is to be used “sparingly”, and that users should prefer using Flatpak or toolbx to access additional packages.

RHEL “Image Mode” (`bootc`) #

bootc based distributions use an alternate approach, packaging the base system into OCI containers (commonly referred to as Docker containers). Atomicity and transactionality are achieved by using container images to deliver the entire core system, and rebooting into a new revision.

RHEL Image Mode uses bootc. This technology capitalises on the success of OCI containers as a transport and delivery mechanism for software by packing an entire OS base image into a single container, including the kernel image.

The bootc project builds on ostree , but where ostree never delivered an opinionated “install mechanism”, bootc does. The contents of a bootc image is an ostree filesystem.

Installing new system packages generally means building a new base image, downloading that image and rebooting into it with a command such as bootc switch <image reference>.

Users often use Flatpak to install graphical user applications atop the immutable base, or a user-space package manager such as brew for other utilities.

NixOS #

The Nix project first appeared in 2003. NixOS is built on top of the Nix package manager, using it to manage both packages and system configuration.

NixOS defines the entire system through a declarative configuration, with changes applied via “generations” that can be rolled back. Changes to the system are applied by “rebuilding” the system configuration, which produces a new “generation”.

Nix packages, and therefore NixOS, eschews the traditional Unix FHS in favour of the Nix “store” and a collection of symlinks and wrappers managed by Nix. Only the Nix package manager can write to the store.

The Nix store also (mostly) enables the building and switching of generations without a reboot. Updates are atomic: new generations must build completely before they can be activated. The home-manager project extends these concepts to the user environment and dotfile management.

The impermanence project requires that every persistent directory is explicitly labelled, or else it’s deleted on every reboot, forcing the base OS to be rebuilt from the Nix store and system configuration - essentially “enforcing” core system immutability between reboots. This was inspired by blog posts “Erase Your Darlings” and “NixOS tmpfs as root”, which are worth a read, too!

Ubuntu Core #

Ubuntu Core achieves immutability by packaging every component (kernel, base system, applications) as Snaps.

Snap confinement enforces isolation, and snapd manages transactional updates and rollbacks. The system is designed for reliability, fleet management, and modular upgrades, making it well-suited for IoT and soon, desktop use.

The key components of an Ubuntu Core deployment are:

Gadget snap: provides boot assets, including board specific binaries and data (bootloader, device tree, etc.)
Kernel snap: kernel image and associated modules, along with initial ramdisk for system initialisation
Base snap: execution environment in which applications run - includes “base” Ubuntu LTS packages
System snaps: packages critical to system function such as Network-Manager, bluez, pulseaudio, etc.
Application snaps: define the functionality of the system, confined to a sandbox
Snapd: manages updates, rollbacks and snapshotting/restoring of user data

In a Core Desktop installation, the desktop environment (GNOME, Plasma, etc.), display manager, login manager would all be delivered as “system snaps”.

Snap confinement ensures packages cannot incorrectly interact with the underlying system or user data without explicit approval. In an Ubuntu Core deployment, this notion is extended to every component of the OS, offering a straightforward yet powerful way to manage risk for each system component.

Summary #

Immutable Linux distributions approach the immutability paradox differently. We explored four different approaches here, and you can learn more about other approaches taken by the likes of SUSE MicroOS (filesystem based immutability) and Vanilla OS (uses ABRoot) in this excellent blog post.

Ubuntu Core focuses on transactional packaging and a clean separation of system & user data. bootc-based systems take a full image-based approach, while NixOS offers extreme flexibility through declarative configuration, but at the cost of complexity.

If you’ve yet to try an immutable Linux distribution I’d recommend giving it a go. Whether you prioritise simplicity, security or declarative control there’s almost certainly an immutable Linux distribution that fits your needs.

Workstation VMs with LXD & Multipass

Tue, 25 Jun 2024 00:00:00 +0000

Introduction #

Over the years, I’ve used countless tools for creating virtual machines - often just for short periods of time when testing new software, trying out a new desktop environment, or creating a more isolated development environment. I’ve gone from just using the venerable qemu at the command line, to full-blown desktop applications like Virtualbox, to using virt-manager with libvirt.

When I joined Canonical back in March 2021, I’d hardly used LXD, and I hadn’t ever used Multipass. Since then, they’ve both become indispensable parts of my workflow, so I thought I’d share why I like them, and how I use each of them in my day to day work.

I work for Canonical, and am therefore invested in the success of their products, but at the time of writing I’m not responsible for either LXD or Multipass, and this post represents my honest opinions as a user of the products, and nothing more.

Installation / Distribution #

Both LXD and Multipass are available as snap packages, and that’s the most supported and recommended route for installation. LXD is available in the repos of a few other Linux distributions (including NixOS, Arch Linux), but the snap package also works great on Arch, Fedora, etc. I personally ran Multipass and LXD as snaps on Arch Linux for a couple of years without issue.

If you’d like to follow along with the commands in this post, you can get setup like so:

1
2
3
4
5
6
7
8
9


sudo snap install lxd
sudo lxd init --minimal

# If you'd like to use LXD/LXC commands without sudo
# run the following command and logout/login:
#
# sudo usermod -aG lxd $USER

sudo snap install multipass

Early on in my journey with NixOS, I packaged Multipass for Nix. I still maintain (and use!) the NixOS module. This was my first ever contribution to NixOS – a fairly colourful review process to say the least…

The result is that you can use something like the following in your configuration, and have multipass be available to you after a nixos-rebuild switch:

1
2
3


{
 virtualisation.multipass.enable = true;
}

LXD has been maintained in NixOS for many years now - and around this time last year I added support for the LXD UI. The screenshots you see throughout this post are all from LXD UI running on a NixOS machine using the following configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


{
 virtualisation.lxd = {
 enable = true;
 zfsSupport = true;
 ui.enable = true;
 };

 networking = {
 firewall = {
 trustedInterfaces = [ "lxdbr0" ];
 };
 };
}

Ubuntu on-demand with Multipass #

Multipass is designed to provide simple on-demand access to Ubuntu VMs from any workstation - whether that workstation is running Linux, macOS or Windows. It is designed to replicate, in a lightweight way, the experience of provisioning a simple Ubuntu VM on a cloud.

Multipass makes use of whichever the most appropriate hypervisor is on a given platform. On Linux, it can use QEMU, LXD or libvirt as backends, on Windows it can use Hyper-V or Virtualbox, and on macOS it can use QEMU or Virtualbox. Multipass refers to these backends as drivers.

Multipass’ scope is relatively limited, but in my opinion that’s what makes it so delightful to use. Once installed, the basic operation of Multipass couldn’t be simpler:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


❯ multipass shell
Launched: primary
Mounted '/home/jon' into 'primary:Home'
Welcome to Ubuntu 24.04 LTS (GNU/Linux 6.8.0-35-generic x86_64)

 * Documentation: https://help.ubuntu.com
 * Management: https://landscape.canonical.com
 * Support: https://ubuntu.com/pro

 System information as of Tue Jun 25 11:17:55 BST 2024

 System load: 0.4 Processes: 132
 Usage of /: 38.9% of 3.80GB Users logged in: 0
 Memory usage: 31% IPv4 address for ens3: 10.93.253.20
 Swap usage: 0%


Expanded Security Maintenance for Applications is not enabled.

3 updates can be applied immediately.
1 of these updates is a standard security update.
To see these additional updates run: apt list --upgradable

Enable ESM Apps to receive additional future security updates.
See https://ubuntu.com/esm or run: sudo pro status


ubuntu@primary:~$

This one command will take care of creating the primary instance if it doesn’t already exist, start the instance and drop you into a bash shell - normally in under a minute.

Multipass has a neat trick: it bundles a reverse SSHFS server that enables easy mounting of the host’s home directory into the VM. This happens by default for the primary instance. As a result the instance I created above has my home directory mounted at /home/ubuntu/Home - making it trivial to jump between editing code/files on my host and in the VM. I find this really useful - I can edit files on my workstation in my own editor, using my Yubikey to sign and push commits without having to worry about complicated provisioning or passthrough to the VM, and any files resulting from a build process on my workstation are instantly available in the VM for testing.

Multipass instances can be customised a little. You won’t find complicated features like PCI-passthrough, but basic parameters can be tweaked. The commands I usually run for setting up a development machine when I’m working on Juju/Charms are:

1
2
3
4
5
6


# Create a machine named 'dev' with 16 cores, 40GiB RAM and 100GiB disk
multipass launch noble -n dev -c 16 -m 40G -d 100G
# Mount my home directory into the VM
multipass mount /home/jon dev:/home/ubuntu/Home
# Get a shell in the VM
multipass shell dev

Once you’re done with an instance, you can remove it like so:

1
2


multipass remove dev
multipass purge

Multipass does have some more interesting features, though most of my usage is represented above. One feature that might be of more interest for MacOS or Windows users is aliases. This feature enables you to alias local commands to their counterparts in a Multipass VM, meaning for example that every time you run docker on your Mac, the command is actually executed inside the Multipass VM:

1
2
3


# Example of mapping the local `mdocker` command -> `docker` in
# the multipass VM
multipass alias dev:docker mdocker

Multipass will launch the latest Ubuntu LTS by default, but there are a number of other images available - including some “appliance” images for applications like Nextcloud, Mosquitto, etc.

There is also the concept of Blueprints which are essentially recipes for virtual machines with a given purpose. These are curated partly by the Multipass team, and partly by the community. A blueprint enables the author to specify cores, memory, disk, cloud-init data, aliases, health checks and more. The recipes themselves are maintained on Github, and you can see the list of available images/blueprints using multipass find:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


❯ multipass find
Image Aliases Version Description
core core16 20200818 Ubuntu Core 16
core18 20211124 Ubuntu Core 18
core20 20230119 Ubuntu Core 20
core22 20230717 Ubuntu Core 22
20.04 focal 20240612 Ubuntu 20.04 LTS
22.04 jammy 20240614 Ubuntu 22.04 LTS
23.10 mantic 20240619 Ubuntu 23.10
24.04 noble,lts 20240622 Ubuntu 24.04 LTS
daily:24.10 oracular,devel 20240622 Ubuntu 24.10
appliance:adguard-home 20200812 Ubuntu AdGuard Home Appliance
appliance:mosquitto 20200812 Ubuntu Mosquitto Appliance
appliance:nextcloud 20200812 Ubuntu Nextcloud Appliance
appliance:openhab 20200812 Ubuntu openHAB Home Appliance
appliance:plexmediaserver 20200812 Ubuntu Plex Media Server Appliance

Blueprint Aliases Version Description
anbox-cloud-appliance latest Anbox Cloud Appliance
charm-dev latest A development and testing environment for charmers
docker 0.4 A Docker environment with Portainer and related tools
jellyfin latest Jellyfin is a Free Software Media System that puts you in control of managing and streaming your media.
minikube latest minikube is local Kubernetes
ros-noetic 0.1 A development and testing environment for ROS Noetic.
ros2-humble 0.1 A development and testing environment for ROS 2 Humble.

The team also recently introduced the ability to snapshot virtual machines, though I must confess I’ve not tried it out in anger yet.

LXD… for VMs? #

For many people, LXD is a container manager - and indeed for many years it could “only” manage containers. LXD was built for running “system containers”, as opposed to “application containers” like Docker/Podman (or Kubernetes). Running a container with LXD is more similar to to running a container with systemd-nspawn, but with the added bonus that it can cluster across machines, authenticate against different identity backends, and manage more sophisticated storage.

Because LXD manages system containers, each container gets its own systemd, and behaves more like a ’lightweight VM’ sharing the host’s kernel. This turns out to be a very interesting property for people who want to get some of the benefits of containerisation (i.e. higher workload density, easier snapshotting, migration, etc.) with more legacy applications that might struggle to run effectively in application containers.

But this post is about virtual machines. Since the 4.0 LTS release, LXD has also supported running VMs with qemu. The API for launching a container is identical to launching a virtual machine. Better still, Canonical provides images for lots of different Linux distributions, and even desktop variants of some images - meaning you can quickly get up and running with a wide range of distributions, for example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# Launch a Ubuntu 24.04 LTS VM
lxc launch ubuntu:noble ubuntu --vm
# Get a shell inside the VM
lxc exec ubuntu bash

# Launch a Fedora 40 VM
lxc launch images:fedora/40 fedora --vm
# Get a shell inside the VM
lxc exec fedora bash

# Launch an Arch Linux VM (doesn't support secure boot yet)
lxc launch images:archlinux arch --vm -c security.secureboot=false
# Get a shell inside the VM
lxc exec arch bash

You can get a full list of virtual machine images like so:

1

lxc image ls images: --format=compact | grep VIRTUAL-MACHINE

LXD Desktop VMs #

Another neat trick for LXD is desktop virtual machines. These are launched with curated images that drop you into a minimal desktop environment that’s configured to automatically login. This has to be one of my favourite features of LXD!

1
2


# Launch a Ubuntu 24.04 LTS desktop VM and get a console
lxc launch images:ubuntu/24.04/desktop ubuntu --vm --console=vga

The guest is pre-configured to work correctly with SPICE, so that means clipboard integration, automatic resizing with the viewer window, USB redirection, etc. The same also works for other distros, as before:

1
2
3
4
5
6
7
8


# Launch an Arch desktop VM
lxc launch images:archlinux/desktop-gnome arch --vm \
 -c limits.cpu=8 \
 -c limits.memory=16GiB \
 -c security.secureboot=false

# Get a console using a separate command (if preferred!)
lxc console --type=vga arch

LXD UI 😍 #

Back in June 2023, Canonical announced early access to the LXD graphical user interface on their blog. The LXD UI is now generally available and enabled by default from LXD 5.21 onwards - though you can find instructions for enabling it on earlier versions in the docs. The summary is:

1
2
3
4


lxc config set core.https_address :8443

sudo snap set lxd ui.enable=true
sudo systemctl reload snap.lxd.daemon

In my opinion, the LXD UI is one of the best, if not the best way to interact with a hypervisor yet. Being a full-stack web application, it gains independence from different GUI toolkits on Linux and, provided the cluster is remote, can be accessed the same way from Windows, Mac and Linux.

I’ve used other hypervisors with web UIs, particularly Proxmox, and I’ve found the experience with LXD UI to be very smooth, even from the early days. The UI can walk you through the creation and management of VMs, containers, storage and networking. The UI can also give you a nice concise summary of each instance (below is the summary of the VM created using the command in the last section):

One of my favourite features is the web-based SPICE console for desktop VMs, which combined with the management features makes it trivial to stand up a desktop VM and start testing:

Why both? #

By now you’ve probably realised that LXD can do everything Multipass can do, and give much more flexibility - and that’s true. LXD is a full-featured hypervisor which supports much more sophisticated networking, PCI-passthrough, clustering, integration with enterprise identity providers, observability through Prometheus metrics and Loki log-forwarding, etc.

Multipass is small, lean and very easy to configure. If I just want a quick command-line only Ubuntu VM to play with, I still find multipass shell to be most convenient - especially with the automatic home directory mounting.

When I want to work with desktop VMs, interact with non-Ubuntu distributions, or work more closely with hardware, then I use LXD. I was already a bit of a closet LXD fan, having previously described it as a bit of a “secret weapon” for Canonical, but since the introduction of the LXD UI, I’m a fully paid up member of the LXD fan club 😉

Summary #

As I mentioned in the opening paragraphs - both LXD and Multipass have become central to a lot of my technical workflows. The reason I packaged Multipass for NixOS, was that I wanted to dive into daily-driving NixOS, but not without Multipass! In my opinion, the LXD UI is one of the most polished experiences for managing containers and VMs on Linux, and I’m really excited for what that team cooks up next.

Secure Boot & TPM-backed Full Disk Encryption on NixOS

Sat, 20 Apr 2024 00:00:00 +0000

Introduction #

For the last decade (whoa…) or so, I’ve defaulted to using LUKS-encrypted drives for my machines. In general, I configure an unencrypted boot/EFI partition, then place either an ext4 or btrfs filesystem inside a LUKS container which is used for the root partition.

Some of my machines also have extra disks: my desktop has a 1TB NVMe drive for root, and a 2TB NVMe “data” drive mounted in my home directory under /home/jon/data. I don’t like having to type two different encryption passphrases at boot, so I usually have the extra disk automatically unlocked by putting the key in a file on the root drive, and placing an entry in /etc/crypttab.

This setup works fine for desktop machines, but it’s cumbersome on headless machines because unattended reboots require the disk passphrase to be entered at boot. Even then, all of my computers are exclusively used by me, and this setup means I have to enter two passwords on every boot to get to a working desktop environment (one for the disk, and one for the login manager).

I solved this recently with a combination of Secure Boot and a Trusted Platform Module (TPM), so let’s look at those first with a brief and high-level overview of each.

What’s a TPM? #

Most machines that have been manufactured in the last decade, and certainly in the last 5 years, contain a cryptographic coprocessor conforming to the Trusted Platform Module (TPM) spec. The TPM is a dedicated microcontroller primarily used for verifying the integrity of a machine.

TPMs can be used for storing cryptographic key material and performing basic cryptographic operations. The general premise is that keys can be loaded into the TPM, which enables the TPM to perform cryptographic operations using that key (signing, encrypting, etc.), but the key cannot be recovered or read from the TPM unless certain conditions are met. TPMs also provide other facilities such as secure random number generation, which in turn enables them to securely generate cryptographic keys.

Verifying system integrity essentially boils down to being able to ensure the machine hasn’t been tampered with between boots, and that the boot process itself hasn’t been compromised. A given firmware or operating system can take hardware “measurements” and store those measurements in dedicated slots called Platform Configuration Registers (PCRs). The measurements pertain to the underlying hardware and configuration of the machine. The TPM itself never performs the actual verification of the PCRs, and in fact has no knowledge of whether a measurement is inherently “good” or “bad”, but it can provide signed attestations of their values, which are then judged by the application requesting the attestation according to some policy.

Each PCR contains a hash representing a particular hardware measurement, which can be read at any time, but cannot be overwritten. Rather than allowing a traditional write operation, PCRs are updated through an “extend” operation which depends on the previous hash value, creating a chain of trust not dissimilar from how a blockchain is formed. This means that a given measurement can never be fully removed from the TPM.

Once the measurements are stored in the PCRs, there are various times and purposes for which the firmware or an operating system might read them - one example is for remote attestation during login to a system. In this scenario the attestation can be used to verify that the machine hasn’t been tampered with (perhaps in an Evil Maid attack). One could also store a Certificate Authority signing key in a TPM, and have the Certificate Authority software interface with the TPM to sign certificates using the PKCS#11 standard.

My use-case is to enabling the TPM to provide the passphrase to unlock a LUKS-encrypted disk, which is what I’ll focus on in this post.

Secure Boot #

Secure Boot is the mechanism by which the code executed by a machine’s Unified Extensible Firmware Interface (UEFI) can be verified as trusted. In the vast majority of cases, the first thing executed by the UEFI is a bootloader.

When Secure Boot is enabled, each binary executed by the UEFI must contain a checksum and a signature - which the UEFI verifies before launching the code. In the case that either the checksum or signature do not match, the UEFI will refuse the execute the code, and the boot process will halt.

Many OEM machines ship with Microsoft Windows installed, and thus ship with the necessary keys to validate signatures created with Microsoft’s certificate authority. Linux systems are able to utilise these keys through shim - a small and easily verifiable piece of software which is signed by Microsoft. shim sits between the UEFI and the bootloader in the boot process, obviating the need for every Linux bootloader to be signed by Microsoft on every release. The shim is designed to extend trust from the keys trusted by the computer’s firmware to a new set of keys controlled by the operating system.

But what does Secure Boot get us in reality? By signing the kernel, and in some cases a single UEFI PE binary known as a Unified Kernel Image (UKI) (which contains the bootloader, the kernel, the command-line used to boot the kernel, and other resources), one can be reasonably sure that the boot process hasn’t been tampered with.

This process thwarts a number of common physical attack vectors, such as manipulating the kernel command line to bypass the machine’s login and drop straight to a root shell - and combined with disk encryption can prevent offline data transfer from the machine. It also defends against malware which compromises the operating system’s boot process such that it can start before the OS and obfuscate it’s presence.

Threat Modelling #

As with any security measure, Secure Boot is not a silver bullet. You should always consider your own personal threat model, and the sorts of attacks you’re looking to defend against.

For example, Secure Boot can help prevent the sort of malware infections described in the previous section, but if an attacker gets physical access to your machine, and the UEFI isn’t adequately protected, they could simply disable secure boot and carry on unhindered.

I mitigate this by password protecting the UEFI. This isn’t perfect, but is likely sufficient protection for my threat model which is more about protecting chancers and petty thieves from gaining access to my information, than from determined attackers who gain physical access to my property.

Storing keys in a TPM is theoretically safe, in that each TPM has a unique seed which cannot be retrieved, and enables the TPM to deterministically generate keys between reboots. It’s very difficult to retrieve the seed, and thus very difficult to duplicate a TPM, but not impossible. Even then, the Linux kernel’s communication with the TPM on-the-wire is unencrypted, and the same can be said for many other subsystems which use the TPM. A recent example of this vulnerability was demonstrated by sniffing a Bitlocker key off the LPC bus in a Lenovo X1 Carbon laptop (using a Raspberry Pi Pico, no less). In many modern machines this is mitigated by the TPM being on-CPU, but the point still stands.

I choose to enroll Microsoft’s platform keys, which in theory degrades the security of my device in the case that Microsoft’s signing key is compromised, though all of my machines are compatible with fwupd and can receive updates to the database through that mechanism if required (and in fact have done in the past 18 months). This could be further mitigated by using custom keys and certificates for the full chain, but this is more overhead for daily operations and updates. It’s also worth considering whether you have the resources to fully secure your own chain - especially by comparison to Microsoft who spend tens of millions of dollars per year on security. If an attacker wants your information, and are able to compromise Microsoft’s CA, your own CA may not be such a hurdle.

Security measures are always a trade-off between Confidentiality, Availability and Integrity (CIA). In general, the more rigidly secure boot is implemented and configured, the more you’re protecting confidentiality and integrity. The choices I’ve made are slightly more in favour of availability, but nonetheless raise the bar for any attacker significantly.

Enabling Secure Boot on NixOS #

Now for the fun part! The process for enabling Secure Boot on NixOS has simplified in recent months owing to the creation of lanzaboote - a project which takes of preparing and signing Unified Kernel Images containing a custom stub, the bootloader, the Linux kernel, the kernel’s initrd and the kernel command line. lanzaboote also takes care of installing the UKI on the ESP partition so the UEFI can execute it at boot.

The lanzaboote stub differs slightly from systemd-stub, in that it doesn’t require the kernel and initrd to be part of the UKI. This is important for a generation-based operating system like NixOS because bundling the kernel and initrd into a new UKI for every generation would consume a lot of disk space, and quickly exhaust the ESP on most machines. In lanzaboote’s implementation, the kernel and initrd are stored separately on the ESP, and the chain of trust is preserved by validating the signature of the kernel, and embedding a cryptographic hash of the initrd into the signed UKI.

The project takes advantage of systems that have bootspec enabled, which is a relatively recent NixOS RFC that ensures configured machines maintain a file containing a set of memoised facts about a system’s closure. Bootspec aims to “provide more uniform feature support” to bootloaders in the NixOS ecosystem and “enable NixOS users to implement custom bootloader tools and policy” - of which lanzaboote is one.

Generating Secure Boot Keys #

The first step is to generate some keys for the secure boot process. This can be achieved using the sbctl package:

1
2
3
4


❯ sudo nix run nixpkgs#sbctl create-keys
Created Owner UUID 6ac34cc3-a23d-9745-ef33-a03f523d20a3
Creating secure boot keys...✓
Secure boot keys created!

This should only take a few seconds at maximum, and will result in a set of keys being populated in /etc/secureboot:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


❯ tree /etc/secureboot
/etc/secureboot
├── files.db
├── GUID
└── keys
 ├── db
 │ ├── db.key
 │ └── db.pem
 ├── dbx
 │ ├── dbx.key
 │ └── dbx.pem
 ├── KEK
 │ ├── KEK.key
 │ └── KEK.pem
 └── PK
 ├── PK.key
 └── PK.pem

Enable Bootspec #

Ensure that bootspec is enabled in your Nix configuration. You can see this in my flake for my desktop machine on Github:

1
2
3


{
 boot.bootspec.enabled = true;
}

Enable `lanzaboote` #

I use a flake to configure all of my machines, so I’m able to get access to lanzaboote by adding the upstream flake as an input to my own nixos-config flake:

1
2
3
4
5


# ...
inputs = {
 lanzaboote.url = "github:nix-community/lanzaboote";
};
# ...

Once you’ve added lanzaboote as a dependency, you’ll need to import the lanzaboote module:

1
2
3


# ...
imports = [ lanzaboote.nixosModules.lanzaboote ];
# ...

In my flake, I use a custom helper function to build NixOS configurations, so the module is passed directly to lib.nixosSystem through the modules attribute.

The lanzaboote module replaces the systemd-boot module, and as such you must explicitly disable systemd-boot when enabling lanzaboote. Additionally, if you wish to use the TPM for disk unlock (described in the next section), you must use the systemd initrd hooks (or something like clevis):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


boot = {
 initrd.systemd.enable = true;

 loader.systemd-boot.enable = lib.mkForce false;

 lanzaboote = {
 enable = true;
 pkiBundle = "/etc/secureboot";
 };
};

This is represented in my config here.

Once enabled, rebuild your system (in my case with sudo nixos-rebuild switch --flake /home/jon/nixos-config) and verify that your machine is ready for Secure Boot. Don’t panic about the kernel images being reported as not signed, this is expected:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


❯ sudo nix run unstable#sbctl verify
Verifying file database and EFI images in /boot...
✓ /boot/EFI/BOOT/BOOTX64.EFI is signed
✓ /boot/EFI/Linux/nixos-generation-414-376jna572gsb23snqs67t7s4bwxzb3epblmdnzweghuepopml2va.efi is signed
✓ /boot/EFI/Linux/nixos-generation-415-iqulgohymbdppgtxzho6ou3fcuxjbxhumpzm4vojmipwy3sbmuna.efi is signed
✓ /boot/EFI/Linux/nixos-generation-416-kxnzioafnduwwck3oypo7rqwtoat745czp2bpehoufp4yqiawypa.efi is signed
✗ /boot/EFI/nixos/kernel-6.8.2-242idodyvf36cpl6s5dskjy6mo4tjhszuwa3hye7qcjyuo5vnehq.efi is not signed
✗ /boot/EFI/nixos/kernel-6.8.5-zqulrwsucm6okcyns6v2jhh6fregk3bvsdth3yloqfymfbgnh64a.efi is not signed
✗ /boot/EFI/nixos/kernel-6.8.7-6mmixkr6ewywm5swgbi5ethbpgnyia4borzmkevcjx7n7t3mtida.efi is not signed
✓ /boot/EFI/systemd/systemd-bootx64.efi is signed

Prepare the UEFI #

Reboot your machine and enter the UEFI interface. This part of the process will vary from machine to machine depending on the UEFI implementation, but you’re looking to enable Secure Boot, and clear the preloaded Secure Boot keys. This may be referred to as “Setup Mode”, or erasing the “Platform Keys”.

While you’re here, I’d also advise setting a UEFI password before rebooting back into NixOS.

Enroll Secure Boot Keys #

The final stage in the process is to enroll your newly generated Secure Boot keys from step 1 into the UEFI. This is again achieved with sbctl:

1
2
3
4


❯ sudo nix run nixpkgs#sbctl enroll-keys -- --microsoft
Enrolling keys to EFI variables...
With vendor keys from microsoft...✓
Enrolled keys to the EFI variables!

I chose to use the --microsoft option to also enroll the UEFI vendor certificates from Microsoft. Some systems contain firmware that is signed and validated when Secure Boot is enabled, and omitting the Microsoft keys could prevent your device from booting - omit this option with caution!

Verify Secure Boot #

Once you’ve enrolled the keys, reboot the machine back into NixOS and use bootctl to confirm that Secure Boot is in fact enabled:

1
2
3
4
5
6
7


❯ bootctl status
System:
 Firmware: UEFI 2.80 (American Megatrends 5.26)
 Firmware Arch: x64
 Secure Boot: enabled (user)
 TPM2 Support: yes
 Boot into FW: supported

TPM Unlock of Root Partition #

Now that we’re (reasonably) confident that no one can tamper with the boot process, we can progress to allowing the machine to auto-unlock the encrypted disk using a key stored in the TPM.

This is actually more common than you might think - Windows has enabled this behaviour by default for some time with Bitlocker disk encryption, and Canonical is also working on bringing TPM-backed full disk encryption to Ubuntu.

This is probably the easiest step of them all! A simple invocation of systemd-cryptenroll is all that’s required. The arguments below instruct the machine that PCRs 0, 2, 7 and 12 should be measured and verified before the TPM is allowed to unlock the disk.

According to the Linux TPM PCR Regsitry, this means the following are measured before the LUKS key is presented:

PCR 0: Core system firmware executable code
PCR 2: Extended or pluggable executable code
PCR 7: SecureBoot state
PCR 12: Kernel command line, system credentials and system configuration images

1

❯ sudo systemd-cryptenroll --tpm2-device=auto --tpm2-pcrs=0+2+7+12 --wipe-slot=tpm2 /dev/nvme0n1p2

And that’s it! The next time you reboot, your disk should be automatically unlocked by the TPM, and your machine should boot straight to your display manager, or the TTY login if no display manager is configured.

Useful Resources #

None of the knowledge in this post is novel, but rather the culmination of some knowledge acquired over the past few years, and some more targeted reading more recently. In the process, I learned a bunch from the following:

Summary #

About 5 years ago, I was sporting a fully secure-boot enabled Dell XPS 13 running Arch Linux. Back then, the process was complicated, manual, and required a lot of maintenance between upgrades. For me, it was more pain than gain, but an interesting learning experience nonetheless.

When I sat down earlier this year to enable Secure Boot on NixOS, I’d set aside a few hours. I was astounded that 10 minutes later I was finished. I wrote this post as a memo to my future self, but also to illustrate how simple it can be to enable Secure Boot and TPM disk unlock in 2024.

I don’t claim to be an expert on the inner workings of TPMs, nor Secure Boot. The things I can say for certain are that TPMs are complex, that there are improvements that could be made to Linux’s interactions with the TPM, and that a determined and well-resourced attacker is likely going to succeed one way or another.

If you spot an inaccuracy in this post, reach out and let me know on Mastodon, on Telegram, by email, or however you prefer!

Until next time!

Linux on Jon Seager

ntpd-rs: it's about time!

Introduction #

NTP, NTS, and PTP #

Proven at Scale #

A Single, Memory-Safe Utility for NTP and PTP #

Timelines and Goals #

About the Trifecta Tech Foundation #

Summary #

An update on upki

Architecture & Progress #

How to try it #

Next Steps #

Further Down the Road #

Certificate Transparency Enforcement #

Intermediate Preloading #

Merkle Tree Certificates #

Summary #

Developing with AI on Ubuntu

Hardware & Drivers #

Inference Snaps #

Sandboxing Agents #

My agent sandboxes itself! #

Sandbox with LXD containers #

Sandbox with LXD VMs #

Sandbox with Multipass #

Sandbox with WSL #

Summary #

Addressing Linux's Missing PKI Infrastructure

The Problem #

CRLite #

Introducing upki #

Ecosystem Compatibility #

Timeline #

Summary #

Ubuntu Summit 25.10: Personal Highlights

DOOM in Space #

Infrastructure-Wide Profiling of Nvidia CUDA #

Inference Snaps #

Nøughty Linux: Ubuntu’s Stability Meets Nixpkgs’ Freshness #

Are we stuck with the same Desktop UX forever? #

Honorable Mentions #

Conclusion #

Ubuntu Engineering in 2025: A Retrospective

Ubuntu 25.10: A Retrospective #

Communication #

Automation #

Process #

Modernisation #

What’s Next? #

The Immutable Linux Paradox

What is an immutable Linux distribution? #

Why immutability? #

The immutability paradox #

Approaches to immutability #

Fedora Silverblue / CoreOS / EndlessOS (ostree) #

RHEL “Image Mode” (bootc) #

NixOS #

Ubuntu Core #

Summary #

Workstation VMs with LXD & Multipass

Introduction #

Installation / Distribution #

Ubuntu on-demand with Multipass #

LXD… for VMs? #

LXD Desktop VMs #

LXD UI 😍 #

Why both? #

Summary #

Secure Boot & TPM-backed Full Disk Encryption on NixOS

Introduction #

What’s a TPM? #

Secure Boot #

Threat Modelling #

Enabling Secure Boot on NixOS #

Generating Secure Boot Keys #

Enable Bootspec #

Enable lanzaboote #

Prepare the UEFI #

Enroll Secure Boot Keys #

Fedora Silverblue / CoreOS / EndlessOS (`ostree`) #

RHEL “Image Mode” (`bootc`) #

Enable `lanzaboote` #