Canonical on Jon Seager

ntpd-rs: it's about time!

Thu, 26 Mar 2026 00:00:00 +0000

Introduction #

I am thrilled to announce the next target in our campaign to replace core system utilities with memory-safe Rust rewrites in Ubuntu. In upcoming releases, Ubuntu will be adopting ntpd-rs as the default time synchronization client and server, eventually replacing chrony, linuxptp and with any luck, gpsd for time syncing use-cases.

ntpd-rs is a full-featured implementation of the Network Time Protocol (NTP), written entirely in Rust. Maintained by the Trifecta Tech Foundation as part of Project Pendulum, ntpd-rs places a strong focus on security, stability, and memory safety.

To deliver on this goal, we’re building on our partnership with the Trifecta Tech Foundation who are behind sudo-rs, zlib-rs and more. We will be funding the Trifecta Tech Foundation to build new features, enhance security isolation, and ultimately deliver a unified, memory-safe time synchronization utility for the Linux ecosystem. This work meshes well with the Trifecta Tech Foundations goals to improve the security of time synchronization everywhere.

NTP, NTS, and PTP #

Before diving into the mechanics and reasoning behind the transition, I wanted to give some background on the protocols at play, and the problems we’re hoping to solve. Keeping accurate time is a critical system function, not least because it involves constant interaction with the internet and forms the basis for cryptographic verification in protocols such as Transport Layer Security (TLS).

NTP (Network Time Protocol) is the foundational protocol that most operating systems implement to accurately determine the current time from a network source.

NTS (Network Time Security) is to NTP what HTTPS is to HTTP. Historically, the Network Time Protocol was used unencrypted, like many of the early web protocols. NTS introduces cryptographic security to time synchronization, ensuring that bad actors cannot intercept or spoof time data. We already pushed to make NTS the default out-of-the-box in Ubuntu 25.10, which we accomplished by migrating away from ntpd to chrony as the default time-syncing implementation.

PTP (Precision Time Protocol) is used for systems that require sub-microsecond synchronization. While the precision offered by a standard NTP deployment is sufficient for general-purpose computing, PTP is often used for complex, specialized deployments like telecommunications networks, power grids, and automotive applications.

Proven at Scale #

Transitioning core utilities in Ubuntu comes with a responsibility to ensure that replacements are of high quality, resilient and offer something to the platform. We may be the first major Linux distribution to adopt ntpd-rs by default, but we aren’t the first to recognize the readiness of ntpd-rs - it has already been proven at scale by Let’s Encrypt.

While Let’s Encrypt’s core Certificate Authority software has always been written in memory-safe Go, their server operating systems and network infrastructure historically relied on memory-unsafe languages like C and C++, which routinely led to vulnerabilities requiring patching.

Following extensive development, ntpd-rs was deployed to Let’s Encrypt’s staging environment in April 2024, and rolled out to full production by June 2024, marking a major milestone for ntpd-rs.

The fact that one of the world’s most prolific and security-conscious certificate authorities trusts ntpd-rs to keep time across its fleet should provide us, and our enterprise customers, with tremendous confidence in its resilience and suitability.

A Single, Memory-Safe Utility for NTP and PTP #

We want to provide a single utility for configuring both NTP/NTS and Precision Time Protocol (PTP) on Linux. The Trifecta Tech Foundation is concurrently developing Statime, a memory-safe PTP implementation that delivers synchronization performance on par with linuxptp, but with the goal of being easier to configure and use.

The goal is to integrate Statime’s PTP capabilities directly into ntpd-rs, improving the user experience by bringing all time synchronization concerns into one utility with common configuration and usage patterns, obviating the need for complex manual configuration (and troubleshooting) that users of linuxptp may be familiar with.

Timelines and Goals #

As with our transition to sudo-rs and uutils coreutils, leading the mainstream adoption of foundational system utilities comes with responsibility. We want to ensure that ntpd-rs matches the security isolation and performance standards our users expect from chrony.

Canonical is funding the Trifecta Tech Foundation’s development efforts toward these goals over the coming cycles. This work will take place between July 2026 and January 2027 in several major milestones. Our current timeline and targeted goals are:

Ubuntu 26.10: If all goes well, we aim to land the latest version of ntpd-rs in the archive, making it available to test.
Ubuntu 27.04: By 27.04, ntpd-rs should have integrated statime, and we will ship the unified client/server binary for NTP, NTS and PTP in Ubuntu by default, with the aim of providing a smooth migration path for those who already manage complex chrony configs.

To get us there, the Trifecta Tech Foundation will be working on the following items:

Feature Parity & Hardware Support: Adding gpsd IP socket support, multi-threading support for NTP servers, and support for multi-homed servers.
Security & Isolation: chrony is isolated via AppArmor and seccomp. We’ll be working on robust AppArmor and seccomp profiles for ntpd-rs to ensure we don’t buy memory safety at the cost of system-level privilege boundaries. We are also ensuring rustls can use openssl as a crypto provider to satisfy strict corporate cryptography policies.
PTP & Automotive Profiles: Adding support for gPTP, which will allow us to support complex deployments like the Automotive profile directly from nptd-rs (via Statime). Additionally, experimental support for the proposed Client-Server PTP protocol (CSPTP, IEEE 1588.1) will be added.
Benchmarking & Testing: Comprehensive benchmarking of long-term memory, CPU usage, and synchronization performance against chrony to give our cloud partners and enterprise users complete confidence in the transition.
User-experience: Logging improvements and enhancements to configuration that help users configure the time synchronisation target to optimise network usage, as well as improvements to the ntp-cli

About the Trifecta Tech Foundation #

Trifecta Tech Foundation is a non-profit and a Public Benefit Organisation (501(c)(3) equivalent) that creates open-source building blocks for critical infrastructure software. Their initiatives on data compression, time synchronization, and privilege boundary, impact the digital security of millions of people. If you’d like to support their work, please contact them via https://trifectatech.org/support.

Summary #

I am really excited to deepen our already productive relationship with the Trifecta Tech Foundation to make these transitions viable for the wider ecosystem. We’ll be working hard on testing and integration to ensure seamless migration paths, and heavily document the changes ahead of the 26.10 and 27.04 releases.

Stay tuned!

An update on upki

Mon, 16 Feb 2026 00:00:00 +0000

Last year, I announced that Canonical had begun supporting the development of upki, a project that will bring browser-grade Public Key Infrastructure (PKI) to Linux. Since then, development has been moving at pace thanks to the tireless work of Dirkjan and Joe.

In this post, I’ll explore the progress we’ve made, how you can try an early version, and where we’re going next.

Architecture & Progress #

As a reminder, upki’s primary goal is to provide a reliable, privacy-preserving, and efficient certificate revocation mechanism for Linux system utilities, package managers, and language runtimes. The solution is built around CRLite, an efficient data format that compresses and distributes certificate revocation information at scale.

The upki repository is structured as a Cargo workspace containing five crates, each serving a distinct role:

upki: the core library and CLI tool. This crate contains the revocation query engine, the client-side sync logic for fetching filter updates, and the command-line interface. The revocation interface was originally embedded in the CLI, but has since been promoted into the library so that other Rust projects can use it directly as a dependency.
upki-mirror: the server-side mirroring tool. This binary fetches and validates CRLite filters from Mozilla’s infrastructure such that they can be served using a standard web server like nginx or apache.
upki-ffi: the C Foreign Function Interface. Built as a cdylib, this crate uses cbindgen to auto-generate a upki.h header file, exposing the revocation query API to C, C++, Go and any other language with C FFI support.
rustls-upki: an integration crate that wires upki’s revocation engine into rustls, enabling any Rust application using rustls to perform CRLite-backed revocation checks transparently.
revoke-test: testing infrastructure for validating revocation queries against known-revoked certificates.

The team recently released v0.1.0, which should help us to gather more feedback on the work we’ve done so far.

How to try it #

If you’d like to try the code in its current form, you’ll need to have a version of the Rust toolchain installed. The easiest way to do this on Ubuntu is using the rustup snap:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# Ensure you have a C compiler in your PATH
sudo apt update
sudo apt install -y build-essential curl

# Install the rustup snap and get the stable toolchain
sudo snap install --classic rustup
rustup install stable

# Install upki
cargo install upki
export PATH="$HOME/.cargo/bin:$PATH"

# Fetch revocation data. This will be done in the background
# when installed through the distro in the future
upki fetch

That should be all you need to install the development version of upki, and you can now use it to run a revocation check by piping certificate output from curl into upki:

1
2


curl -sw '%{certs}' https://google.com | upki revocation check
NotRevoked

Early versions of docs for the C FFI crate and Rust crate documentation are available, but if you’d like to explore, build the project from source, or contribute, the repository is the best place to start. For an example of the C FFI interface in action you can take a look at the upki-go-demo Dirkjan published.

Next Steps #

Now the foundational pieces are in place, our focus is shifting to external consumption, performance, and integration with the wider Linux ecosystem. In the coming days there should be an early 0.1.0 binary release.

We’ll also be doing some performance benchmarking on the initial fetch and of the revocation checks themselves. Currently, each revocation check reads several CRLite filter files into memory. There may be quick wins to improve this, but we’ll benchmark first and see if it warrants optimisation at this time.

We also need to deploy some production infrastructure for serving the CRLite filters. If you follow the steps above, you’ll be fetching from a pre-production web server hosted at https://upki.rustls.dev. We’ve built a Juju charm for operating the CRLite mirror on Kubernetes. This charm packages the upki-mirror binary in a chiselled Rock, and will be deployed into Canonical’s datacentres to serve CRLite data at crlite.ubuntu.com.

Our Ubuntu Foundations team is also working on packaging the various upki components for inclusion in the Ubuntu archive, which will enable you to apt install upki in the future, and also enable us to package and enable it by default in Ubuntu 26.10 and beyond.

Further Down the Road #

While the work above covers what’s immediately in front of us, there is scope to expand upki’s capabilities further. Two areas of interest are Certificate Transparency enforcement, and support for Merkle Tree Certificates.

Certificate Transparency Enforcement #

While upki’s initial focus is on revocation checking, the project also aims to eventually support Certificate Transparency (CT) enforcement. CT is a more modern security measure that relies upon a set of publicly auditable, append-only logs that record every TLS certificate issued by a Certificate Authority (CA). This prevents CAs from issuing fraudulent or erroneous certificates without a means for that fraudulent activity to be discovered - a problem that has bitten organisations in the past.

CT Enforcement would enable clients to refuse to establish a connection unless the server provides cryptographic proof that its certificate has been correctly logged. Browsers like Chrome and Firefox already enforce this, but the rest of the Linux ecosystem would need a tool such as upki to enable such functionality.

Intermediate Preloading #

A correctly configured TLS server should not only send its own certificate, but also the intermediate certificates needed to chain back to a trusted root. In practice, many servers omit the intermediate certificates, and because browsers have quietly worked around this for years, the misconfiguration often goes unnoticed.

Firefox has been preloading all intermediates disclosed to the Common CA Database (CCADB) since Firefox 75, while Chrome and Edge will silently fetch missing intermediates using the Authority Information Access (AIA) extension in the server’s certificate. The result is that a broken certificate chain that works perfectly in every browser will produce an opaque UNKNOWN_ISSUER error when accessed by Linux utilities like curl.

Because upki already maintains a regularly synced local data store, it’s well positioned to ship the known set of intermediates alongside the CRLite filters. This wouldn’t provide a security improvement so much as a usability improvement. It would also bring non-browser clients up to parity with browsers with respect to connection reliability. There is an additional privacy benefit too: rather than fetching a missing intermediate from the issuing CA (which discloses browsing activity to the CA), the intermediate is already present locally.

Merkle Tree Certificates #

Looking even further ahead, upki could support the next generation of web PKI by including support for Merkle Tree Certificates (MTCs). This is an area of active development in the IETF, with Cloudflare and Chrome recently announcing an experimental deployment.

The motivation for MTCs comes largely from the transition to Post-Quantum (PQ) cryptography. PQ signatures are significantly larger than their non-PQ counterparts. The signatures for ML-DSA-44 are 2,420 bytes compared to 64 bytes for ECDSA-P256. A typical TLS handshake today involves multiple signatures and public keys across the certificate chain and CT proofs, which means a simple swap to PQ algorithms would add tens of kilobytes of overhead per connection and likely a noticeable increase in connection latency.

MTCs address this by rethinking how certificates are validated. Rather than transmitting a full certificate chain with multiple signatures, a Certificate Authority can batch certificates into a Merkle Tree and sign only the tree’s root hash. The client then receives just a single signature, a public key, and a compact Merkle tree inclusion proof that demonstrates the certificate’s presence in the batch. The signed tree heads can be distributed to clients out-of-band, meaning the per-handshake overhead is drastically reduced.

Because upki already maintains a local data store that is regularly synced, it could cache tree head data alongside CRLite filters, thereby enabling the inclusion proofs sent during TLS handshakes to be even smaller. Rather than proving inclusion all the way from the leaf to the root, the server could send a “truncated” proof that starts partway up the tree, with the client computing the remainder from data it already has locally. There is a TLS extension being developed to negotiate this.

The implementation of MTCs for TLS is still highly experimental. MTCs are not yet deployed in any browser, but upki will lay the groundwork for Linux system utilities to benefit from this evolution as the technology is adopted.

Summary #

In the few weeks since we announced upki, the core revocation engine has been established and is now functional, the CRLite mirroring tool is working and a production deployment in Canonical’s datacentres is ongoing. We’re now preparing for an alpha release and remain on track for an opt-in preview for Ubuntu 26.04 LTS.

Beyond revocation, we’re keeping a close eye on the evolving PKI landscape and particularly CT enforcement and Merkle Tree Certificates.

I’d like to extend my thanks again to Dirkjan and Joe for their continued collaboration on this work, and the utmost professionalism they’ve demonstrated throughout.

Developing with AI on Ubuntu

Tue, 20 Jan 2026 00:00:00 +0000

AI-assisted tooling is becoming more and more common in the workflows of engineers at all experience levels. As I see it, our challenge is one of consideration, enablement and constraint. We must enable those who opt-in to safely and responsibly harness the power of these tools, while respecting those who do not wish to have their platform defined or overwhelmed by this class of software.

The use of AI is a divisive topic among the tech community. I find myself a little in both camps, somewhere between sceptic and advocate. While I’m quick to acknowledge the negative impacts that the use of LLMs can have on open source projects, I’m also surrounded by examples where it has been used responsibly to great effect.

Examples of this include Filippo’s article debugging low-level cryptography with Claude Code, Mitchell’s article on Vibing a Non-Trivial Ghostty Feature, and David’s article How I Program with Agents. These articles come from engineers with proven expertise in careful, precise software engineering, yet they share an important sentiment: AI-assisted tools can be a remarkable force-multiplier when used in conjunction with their lived experience, but care must still be taken to avoid poor outcomes.

The aim of this post is not to convince you to use AI in your work, but rather to introduce the elements of Ubuntu that make it a first-class platform for safe, efficient experimentation and development. My goals for AI and Ubuntu are currently focused on enabling those who want to develop responsibly with AI tools, without negatively impacting the experience of those who’d prefer not to opt-in.

Hardware & Drivers #

AI-specific silicon is moving just as fast as AI software tooling, and without constant work to integrate drivers and userspace tools into Ubuntu, it would be impossible to efficiently utilise this specialised hardware.

Last year we announced that we will ship both NVIDIA’s CUDA and AMD’s ROCm in the Ubuntu archive for Ubuntu 26.04 LTS, in addition to our previous work on OpenVINO. This will make installing the latest drivers and toolkits easier and more secure, with no third-party software repositories. Distributing this software as part of Ubuntu enables us to be proactive in the delivery of security updates and the demonstration of provenance.

Our work is not limited to AMD and NVIDIA; we recently announced support for Qualcomm’s Dragonwing platforms and others. You can read more about our silicon partner projects on our website.

Inference Snaps #

At the Ubuntu Summit 25.10, we released “Inference Snaps” into the wild, which provide a hassle-free mechanism for obtaining the “famous model” you want to work with, but automatically receive a version of that model which is optimised for the silicon in your machine. This removes the need to spend hours on HuggingFace identifying the correct model to download that matches with your hardware, and obviates the need for in-depth understanding of model quantisation and tuning when getting started.

Each of our inference snaps provide a consistent experience: you need only learn the basics once, but can apply those skills to different models as they emerge, whether you’re on a laptop or a server.

At the time of writing, we’ve published beta quality snaps for qwen-vl, deepseek-r1 and gemma3. You can find a current list of snaps in the documentation, along with the silicon-optimised variants.

Sandboxing Agents #

While many start their journey in a web browser chatting to ChatGPT, Claude, Gemini, Perplexity or one of the myriad of alternatives, many developers will find “agentic” tools such as Copilot, Codex, Claude Code or Amp quite attractive. In my experience, agents are a clear level-up in an LLM’s capability for developers, but they can still make poor decisions and are generally safer to run in sandboxed environment at the time of writing.

Where a traditional chat-based AI tool responds reactively to user prompts within a single conversation, an agent operates (semi-)autonomously to pursue goals. It perceives its environment, plans, makes decisions and can call out to external tools and services to achieve those goals. If you grant permission, an agent can read and understand your code, implement features, troubleshoot bugs, optimise performance and many other tasks. The catch is that they often need access to your system - whether that be to modify files or run commands.

Issues such as accidental file deletion, or the inclusion of a spurious (and potentially compromised) dependency are an inevitable failure mode of the current generation of agents due to how they’re trained (see the Reddit post about Claude Code deleting a user’s home directory).

My agent sandboxes itself! #

Some of you will be reading this wondering why additional sandboxing is required, since many of the popular agents advertise their own sandboxing. The fact that some agents include some measures to protect the user’s machine is of course a good thing. The touted benefits include filesystem isolation by restricting the agent to a specific directory, or prompting for approval before modifying files. Some agents also include network sandboxing to restrict network access to a list of approved domains, or by using a custom proxy to impose rules on outbound traffic.

On Linux, these agent-imposed sandboxes are often implemented with bubblewrap, which is “a tool for constructing sandbox environments”, but note that the upstream project’s README includes a section which states that it is not a “ready-made sandbox with a specific security policy”. bubblewrap is a relatively low-level tool that must be given its configuration, which in this case is provided by the agent.

The limitation upon these tools is the shared kernel - a severe kernel exploit could enable an agent to escape from its sandbox. Of course, such vulnerabilities are rare, but note that even if the sandboxing technologies do their job, agents often run in the context of the user’s session, meaning they inherit environment variables which could contain sensitive information. They’re also agent specific: Claude Code’s sandboxing won’t help you if you’re using Cursor or Antigravity.

Depending on your threat model and the project you’re working on, you may deem the built-in sandboxing of coding agents to be sufficient, but there are other options available to Ubuntu users that provide either different, or additional protection…

Sandbox with LXD containers #

Canonical’s LXD works out-of-the-box on Ubuntu, and is a great way to sandbox an agent into a disposable environment where the blast radius is limited should the agent make a mistake. My personal workflow is to create an Ubuntu container (or VM) with my project directory mounted. This way, I can edit my code directly on my filesystem with my preferred (already configured) editor, but have the agent run inside the container.

For example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Initialise the container
lxc init ubuntu:noble dev
# Mount my project directory into the container
lxc config device add -q dev datadir disk source="$HOME/my-project" path=/home/ubuntu/project
# Start the container
lxc start dev
# Get a shell inside the container as the 'ubuntu' user
lxc exec dev -- sudo -u ubuntu -i bash
# Run a command in the container
lxc exec dev -- sudo -u ubuntu -i bash -c "cd project; claude"

You can learn more about LXD in the official documentation and tutorial, as well as specific instructions on enabling GPU data processing in containers/VMs. I’ve written previously about my use of LXD in development.

With LXD, you can choose between running your sandbox as a container or a VM, depending on your project’s needs. If I’m working on a project that requires Kubernetes or similar, I use a VM, but for lighter projects I use system containers, preferring their lower overhead.

Sandbox with LXD VMs #

LXD is best known for its ability to run “system containers”, which are somewhat analogous to Docker/OCI containers, but rather than being focused on a single application (and dependencies), a system container essentially runs an entire Ubuntu user-space (including systemd, etc.). Like OCI containers, however, system containers share the kernel with the host.

In some situations, you may seek more isolation from your host machine by running tools inside a virtual machine with their own kernel. LXD makes this simple - you can follow the same commands as above, but add --vm to the init command:

1
2


# Initialise the virtual machine
lxc init --vm ubuntu:noble dev

You can also configure the virtual machine’s CPU, memory and disk requirements. A simple example is below:

1
2
3
4


lxc init --vm ubuntu:noble dev \
 -c limits.cpu=8 \
 -c limits.memory=8GiB \
 -d root,size=100GiB

You can find more details on instance configuration in the LXD documentation.

Sandbox with Multipass #

Multipass provides on-demand access to Ubuntu VMs from any workstation - whether that workstation is running Linux, macOS or Windows. It is designed to replicate, in a lightweight way, the experience of provisioning a simple Ubuntu VM on a cloud.

Multipass’ scope is more limited than LXD, but for many users it provides a simple on-ramp for development with Ubuntu. Where it lacks advanced features like GPU passthrough, it boasts a simplified CLI and a first-class GUI client.

To get started similarly to the LXD example above, try the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Install Multipass
sudo snap install multipass
# Launch an instance
multipass launch noble -n dev
# Mount your project directory
multipass mount ~/my-project dev:/home/ubuntu/project
# Get a shell in the instance
multipass shell dev
# Run a command in the instance
multipass exec dev -- claude

You can find more details on how to configure and manage instances in the docs.

Sandbox with WSL #

If you’re on Windows, development with WSL includes first-class support for GPU acceleration, and is even supported for use with the NVIDIA AI Workbench, NVIDIA NIM and CUDA.

Ubuntu is the default Linux distribution for WSL, and you can find more information about how to set up and use Ubuntu on WSL in our documentation. WSL benefits from all the same technologies as a “regular” Ubuntu install, including the ability to use Snaps, Docker and LXD.

For the enterprise developer, we recently announced Ubuntu Pro for WSL, as well as the ability to manage WSL instances using Landscape, making it easier to get access to first-class developer tooling with Ubuntu on your corporate machine.

Summary #

While opinion remains divided on the value and impact of current AI tooling, its presence in modern development workflows and its demands on underlying compute infrastructure are difficult to ignore.

Developers who wish to experiment need reliable access to modern hardware, predictable tooling, and strong isolation boundaries. Ubuntu’s role is not to dictate how these tools are used, but to provide a stable and dependable platform on which they can be explored and deployed safely, without compromising security, provenance, or the day-to-day experience of those who choose to opt out.

In addition to powering development workflows, Ubuntu makes for a dependable production operating system for your workloads. We’re building Canonical Kubernetes with first-class GPU support, Kubeflow and MLFlow for model training and serving and a suite of applications like PostgreSQL, MySQL, Opensearch, as well as other data-centric tools such as Kafka and Spark that can be deployed with full Ubuntu Pro support. Let me know if you’d find value in a follow-up post on those topics!

Addressing Linux's Missing PKI Infrastructure

Mon, 08 Dec 2025 00:00:00 +0000

Earlier this year, LWN featured an excellent article titled “Linux’s missing CRL infrastructure”. The article highlighted a number of key issues surrounding traditional Public Key Infrastructure (PKI), but critically noted how even the available measures are effectively ignored by the majority of system-level software on Linux.

One of the motivators for the discussion is that the Online Certificate Status Protocol (OCSP) will cease to be supported by Let’s Encrypt. The remaining alternative is to use Certificate Revocation Lists (CRLs), yet there is little or no support for managing (or even querying) these lists in most Linux system utilities.

To solve this, I’m happy to share that in partnership with rustls maintainers Dirkjan Ochtman and Joe Birr-Pixton, we’re starting the development of upki: a universal PKI tool. This project initially aims to close the revocation gap through the combination of a new system utility and eventual library support for common TLS/SSL libraries such as OpenSSL, GnuTLS and rustls.

The Problem #

Online Certificate Authorities responsible for issuing TLS certificates have long had mechanisms for revoking known bad certificates. What constitutes a known bad certificate varies, but generally it means a certificate was issued either in error, or by a malicious actor of some form. There have been two primary mechanisms for this revocation: Certificate Revocation Lists (CRLs) and the Online Certificate Status Protocol (OCSP).

In July 2024, Let’s Encrypt announced the deprecation of support for the Online Certificate Status Protocol (OCSP). This wasn’t entirely unexpected - the protocol has suffered from privacy defects which leak the browsing habits of users to Certificate Authorities. Various implementations have also suffered reliability issues that forced most implementers to adopt “soft-fail” policies, rendering the checks largely ineffective.

The deprecation of OCSP leaves us with CRLs. Both Windows and macOS rely on operating system components to centralise the fetching and parsing of CRLs, but Linux has traditionally delegated this responsibility to individual applications. This is done most effectively in browsers such as Mozilla Firefox, Google Chrome and Chromium, but this has been achieved with bespoke infrastructure.

However, Linux itself has fallen short by not providing consistent revocation checking infrastructure for the rest of userspace - tools such as curl, system package managers and language runtimes lack a unified mechanism to process this data.

The ideal solution to this problem, which is slowly becoming more prevalent, is to issue short-lived credentials with an expiration of 10 days or less, somewhat removing the need for complicated revocation infrastructure, but reducing certificate lifetimes is happening slowly and requires significant automation.

CRLite #

There are several key challenges with CRLs in practice - the size of the list has grown dramatically as the web has scaled, and one must collate CRLs from all relevant certificate authorities in order to be useful. CRLite was originally proposed by researchers at IEEE S&P and subsequently adopted in Mozilla Firefox. It offers a pragmatic solution to the problem of distributing large CRL datasets to client machines.

In a recent blog post, Mozilla outlined how their CRLite implementation meant that on average users “downloaded 300kB of revocation data per day, a 4MB snapshot every 45 days and a sequence of “delta-updates” in-between”, which amounts to CRLite being 1000x more bandwidth-efficient than daily CRL downloads.

At its core, CRLite is a data structure compressing the full set of web-PKI revocations into a compact, efficiently queryable form. You can find more information about CRLite’s design and implementation on Mozilla’s Security Blog.

Introducing upki #

Following our work on oxidizing Ubuntu, Dirkjan reached out to me with a proposal to introduce a system-level utility backed by CRLite to non-browser users.

upki will be an open source project, initially packaged for Ubuntu but available to all Linux distributions, and likely portable to other Unix-like operating systems. Written in Rust, upki supports three roles:

Server-side mirroring tool: responsible for downloading and mirroring the CRLite filters provided by Mozilla, enabling us to operate independent CDN infrastructure for CRLite users, and serving them to clients. This will insulate upki from changes in the Mozilla backend, and enable standing up an independent data source if required. The server-side tool will manifest as a service that periodically checks the Mozilla Firefox CRLite filters, downloads and validates the files, and serves them.
Client-side sync tool: run regularly by a systemd-timer, network-up events or similar, this tool ensures the contents of the CDN are reflected in the on-disk filter cache. This will be extremely low on bandwidth and CPU usage assuming everything is up to date.
Client-side query tool: a CLI interface for querying revocation data. This will be useful for monitoring and deployment workflows, as well as for users without a good C FFI.

The latter two roles are served by a single Rust binary that runs in different modes depending on how it is invoked. The server-side tool will be a separate binary, since its use will be much less widespread. Under the hood, all of this will be powered by Rust library crates that can be integrated in other projects via crates.io.

For the initial release, Canonical will stand up the backend infrastructure required to mirror and serve the CRLite data for upki users, though the backend will be configurable. This prevents unbounded load on Mozilla’s infrastructure and ensures long-term stability even if Firefox’s internal formats evolve.

Ecosystem Compatibility #

So far we’ve covered the introduction of a new Rust binary (and crate) for supporting the fetching, serving and querying of CRL data, but that doesn’t provide much service to the existing ecosystem of Linux applications and libraries in the problem statement.

The upki project will also provide a shared object library for a stable ABI that allows C and C-FFI programs to make revocation queries, using the contents of the on-disk filter cache.

Once upki is released and available, work can begin on integrating existing crypto libraries such as OpenSSL, GNUtls and rustls. This will be performed through the shared object library by means of an optional callback mechanism these libraries can use to check the revocation lists before establishing a connection to a given server with a certificate.

Timeline #

While we’ve been discussing this project for a couple of months, ironing out the details of funding and design, work will soon begin on the initial implementation of upki.

Our aim is to make upki available as an opt-in preview for the release of Ubuntu 26.04 LTS, meaning we’ll need to complete the implementation of the server/client functionality, and bootstrap the mirroring/serving infrastructure at Canonical before April 2026.

In the following Ubuntu release cycle, the run up to Ubuntu 26.10, we’ll aim to ship the tool by default on Ubuntu systems, and begin work on integration with the likes of NSS, OpenSSL, GNUtls and rustls.

Summary #

Linux has a clear gap in its handling of revocation data for PKIs. Over the coming months we’re hoping to address that gap by developing upki not just for Ubuntu, but for the entire ecosystem. Thanks to Mozilla’s work on CRLite, and the expertise of Dirkjan and Joe, we’re confident that we’ll deliver a resilient and efficient solution that should make a meaningful contribution to systems security across the web.

If you’d like to do more reading on the subject, I’d recommend the following:

LWN.net: Linux’s missing CRL infrastructure
Mozilla Security Blog: CRLite Part 1: All Web PKI Revocations Compressed
Mozilla Security Blog: CRLite Part 2: End-to-End Design
Let’s Encrypt: Replacing OCSP with CRLs
IEEE Symposium on Security & Privacy: CRLite: A Scalable System for Pushing All TLS Revocations to All Browsers

Ubuntu Summit 25.10: Personal Highlights

Sun, 02 Nov 2025 00:00:00 +0000

I recently had the privilege of attending the Ubuntu Summit 25.10. Ubuntu Summits have a relatively long history. Some years ago Canonical ran the ‘Ubuntu Developer Summits (UDS)’, but recently the events were brought back and reimagined as the ‘Ubuntu Summit’.

For the most recent Summit, we tried out a new format. We invited a select few folks to come and give talks at our London office, with a small in-person crowd. In addition, the event was livestreamed, and we encouraged people to host “watching parties” across the world as part of Ubuntu Local Communities (LoCos).

While Ubuntu may feature in the name, the event does not require talks to be centred on Ubuntu, and in fact is aiming to draw contributions from our partners and from right across the open source community, whether or not the content is relevant to Ubuntu or Canonical - it’s designed to be a showcase for the very best of open source, and this year I felt that the talks were of a particularly high calibre.

In this post I’ll highlight some of my favourite talks, in no particular order! If any of these catch your interest, you can see when they were aired and catch-up on the Day 1 and Day 2 streams.

DOOM in Space #

What a way to kick off the Summit! DOOM in Space was a talk given by Ólafur Waage, who introduced himself as a “professional keyboard typist”!

The talk was immediately after Mark Shuttleworth’s opening remarks, and covered his journey in getting DOOM to run on the European Space Agency’s OPS-SAT satellite. DOOM has famously been ported to many devices, though some were only questionably “running” the game.

Ólafur covered how he became involved in the project, and the unique approach they needed to take to guarantee success, since they would only get a very limited amount of time in order to conduct their “experiment” on the satellite.

Of particular note was the work done to integrate imagery from the OPS-SAT’s onboard camera into the game, which involved some clever reassigning of colors in the game’s original palette to more faithfully represent the imagery taken from the camera in-game.

Infrastructure-Wide Profiling of Nvidia CUDA #

This talk was given by Frederic Branczyk, CEO and Founder of Polar Signals. Canonical has partnered with Polar Signals a couple of times in recent years. They were part of our journey to enabling frame pointers by default on Ubuntu, and many of our teams have been using their zero-instrumentation eBPF profiler.

While CPU profiling has been commonplace for developers for many years, giving the ability to analyse CPU and memory-bound workloads, profiling GPU workloads has been less prominent, and is particularly difficult in production.

Polar Signals advocate for “continuous profiling”, which means running a profiler at all times, on all nodes, in production. The benefit of this is that when an issue occurs, you don’t have to set up a profiler and try to reproduce the issue - you already have the data. It also negates the uncertainty of the impact a profiler might have on the code during reproduction. This would have been difficult with traditional profiling tools, but with technologies like eBPF, the overhead of the profiler is incredibly low compared to the potential performance gains from acting on the data it produces.

In this talk, Frederic outlined the work they have done bringing infrastructure-wide profiling of CUDA workloads into Polar Signals Cloud. Their approach combines the CUPTI profiling API with USDT probes and eBPF into a pipeline, relying upon the ability to inject a small library into CUDA workloads using the CUDA_INJECTION64_PATH without modification.

You can see more details on their website.

Inference Snaps #

This talk served as the first public announcement of Inference Snaps from Canonical, which represents a few months of working combining many of the new technologies behind Snaps.

As Large Language Models continue to gain pace along with the rest of the AI community, silicon manufacturers are increasingly including dedicated hardware in commodity CPUs and GPUs, as well as shipping dedicated accelerators for some workloads.

AI models often need to be tuned in some way in order to work optimally - for example quantisation which aims to reduce the computational memory costs of running inference on a given model.

Inference snaps provide a hassle-free mechanism for users to obtain the “famous model” they want to work with, but automatically receive a version of that model which is optimised for the silicon in their machine, removing the need to spend hours on HuggingFace trying to identify the correct model to download that matches with their hardware.

Using our extensive partner network, we’ll continue to work with multiple silicon vendors to ensure that models are available for the latest hardware as it drops, and provide a consistent experience to Ubuntu users that wish to work with AI.

Find out more in the announcement.

Nøughty Linux: Ubuntu’s Stability Meets Nixpkgs’ Freshness #

This talk was a bit of a guilty pleasure for me! Delivered by Martin Wimpress (wimpy), the audience were shown how they could take a stock Ubuntu Server deployment, and use a collection of scripts to layer a cutting-edge GUI stack on top using Nixpkgs.

Wimpy outlined his motivation as wanting to rely upon the stable kernel and hardware support offered by Ubuntu, but wanting to be more experimental with his desktop environment and utilities - preferring a tiling window management experience.

Having spent some years on NixOS, Wimpy was recently required to run a security “agent” for work, which was very difficult to enable on NixOS, but worked out of the box on Ubuntu. Recognising the need to make the switch, he was reluctant to move away from the workflow he’d built so much muscle-memory around - and so Nøughty Linux was born!

Nøughty Linux is not a Linux distribution, rather a set of configurations for an Ubuntu Server machine. It utilises nix-system-graphics and system-manager and is actually very similar to a configuration I ran in my own nixos-config repository for my laptop for a while - though Wimpy has chased down significantly more of the papercuts than I did!

Are we stuck with the same Desktop UX forever? #

Scott Jenson delivered an incredibly engaging talk in which he posited that desktop user experience has somewhat stagnated, and worse that many of the patterns we’ve become used to on the desktop are antiquated and unergonomic.

The crux of the talk was to focus on user experience, rather than user interfaces - challenging developers to think about how people learn, and how desktops could benefit more from design affordances by rethinking some critical elements such as window management or text editing.

Using his years of experience at Apple, Symbian and Google, Scott delivered one of the most engaging conference talks I’ve seen, and I thoroughly recommend watching it on our YouTube channel!

Honorable Mentions #

In addition to the talks above, it was a delight to meet Mark Schoolderman from the Trifecta Tech Foundation in-person, who led the work on sudo-rs as part of our “Oxidising Ubuntu” story, and interesting to hear about the value the project derived from Ubuntu’s Main Inclusion Review process as part of landing sudo-rs in main.

Equally, I was delighted that Samuele Kaplun from Proton could join us to talk about the work we’ve been doing together on bringing first-class Snap packages for Proton Mail, Proton VPN, Proton Pass and Proton Authenticator to the Snap store, and their reasons for choosing Snaps, adventures with confinement, and more.

I was delighted to see Craig Loewen and Clint Rutkas present on their journey open sourcing the Windows Subsystem For Linux (WSL), which represents a growing proportion of Ubuntu users, and a key bridge to open source development for many.

Finally, thank you to Utkarsh for this wonderful slide as part of his talk on Ubuntu Snapshot Releases:

Conclusion #

Overall, I found the Ubuntu Summit 25.10 a really enjoyable event, with talks that were uniformly high in quality, charisma and creativity. I’m pleased that Canonical has broadened the Summit’s reach and I hope it continues to serve as a platform to showcase the very best open source innovation.

Until next time!

Ubuntu Engineering in 2025: A Retrospective

Thu, 09 Oct 2025 00:00:00 +0000

Ubuntu 25.10: A Retrospective #

In February this year, I published Engineering Ubuntu For The Next 20 Years, which was something of a manifesto I pledged to enact in the design, build and release of Ubuntu. This week, we released Ubuntu 25.10 Questing Quokka, which was the first full engineering cycle under this new manifesto, and it seems like a good time to reflect on what we achieved in each category, as well as highlight some of the more impactful changes that have just landed in Ubuntu.

In that first article, I outline four themes for Ubuntu Engineering at Canonical to focus on: Communication, Automation, Process and Modernisation.

Communication #

A notable improvement throughout this engineering cycle has been the frequency with which the teams at Canonical have written about their work, often in some detail. Many of these posts can be found under the blog tag, which had never been used until around six months ago, and now sees a couple of new posts per week outlining the work people are doing toward these themes.

I stated that I consider documentation a key part of our communication strategy, and this last six months has seen some of the most substantial changes to Ubuntu documentation in many years. The Ubuntu Project Docs was a project started in May 2025, and is quickly becoming the single documentation hub that a current or potential Ubuntu contributor needs to understand how, why and when to do their job. Similarly, the Ubuntu for Developers was created to illuminate a path for developers across numerous languages on Ubuntu.

It’s important for us to celebrate such efforts, but also to remember that this is only the start! In order for these efforts to remain useful, both our internal teams and our community must continue to engage with these efforts - adding, refining and pruning content as necessary. As the sun-setting of wiki.ubuntu.com approaches, it’s imperative that these new documentation sites continue to get the attention they need.

Lots of the changes we’ve made in the last cycle have attracted attention from online blogs, news outlets, youtubers, etc. Part of the challenge with such changes is “owning the narrative” and ensuring that legitimate concerns are heard (and taken into account), but also that there are appropriate responses to uncertainty, without getting drawn into unproductive discussions.

Finally, the transition to Matrix as the default synchronous communication means for the project has, in my opinion, made it easier than ever to get in touch with our community of experts - whether it be for support, or to start a journey for contribution to Ubuntu.

Automation #

The largest item we took on here was in pursuit of the monthly snapshot releases. This went much better than I expected, and to some extent covers off the “Process” theme as well as “Automation”, but through a combination of studying our process and whittling it down as lean as we could, and beginning to automate more of the process, the team were able to release four snapshot releases before the 25.10 Beta.

The scale of the automation efforts was relatively limited this cycle, but the automation of release testing has really accelerated in the past few months. The vast majority of the test cases that qualify an Ubuntu Desktop ISO for release are now fully automated, and the same framework that makes this possible was also used to develop a suite of tests for TPM FDE.

Work was also done on our craft tools to better the experience with the test sub-command of build tools like snapcraft, rockcraft and charmcraft - all of which will have a trickle-down effect on the upcoming debcraft, and make it trivial to include many new kinds of tests in our packaging workflows.

Behind the scenes, every team in Ubuntu Engineering at Canonical has been writing charms that make the underlying infrastructure behind Ubuntu more portable, resilient and scalable. This includes services like Ubuntu Manpages, autopkgtest, error-tracker, and a staging deployment of Temporal to enable the next phase of our release automation.

Process #

This item was probably where the least concrete progress was made, though I probably could have predicted that. Many of the processes in the Ubuntu project serve to ensure that we ship resilient software, and don’t break users - so changing them in a hurry is not generally a good idea.

That said, there was some good progress on the Main Inclusion Review (MIR) process, whose team documentation was moved into the Ubuntu Project Docs after a thorough review, and the Stable Release Updates (SRU) team are in the process of the same transition. Moving and re-reviewing the documentation is essentially the first step of the process improvement I was seeking: understanding where we are!

Internally, we’ve been piloting a new process for onboarding Ubuntu Developers that sees engineers start by working toward gaining upload rights for a single package, but has a complete curriculum that can take them through to Core Developer status. Details of this should be released in the coming months, outlining a clear and well-trodden journey for new contributors. Much of this material already existed, but the team have worked on polishing it, and making it clearer how the process work from end to end.

The next step for each of these processes is measurement. We’ve begun instrumenting these processes to understand where the most time is spent so we can use that information to guide improvements and streamline processes in future cycles, and even set Service Level Objectives (SLOs) against those timelines.

Modernisation #

Much of what I’ve already described could be considered modernisation, but from a technical standpoint the most obvious candidate here was the “Oxidising Ubuntu” effort, which has seen us replace numerous core utilities in Ubuntu 25.10 with modern Rust rewrites.

We began this effort in close collaboration with the uutils project and the Trifecta Tech Foundation. The former is the maintainer of a Rust coreutils rewrite, and the latter the maintainer of sudo-rs, which we made the default in 25.10. The technical impact of these changes in defaults will only truly be known once Ubuntu 25.10 is “out there”, but I’m pleased with how we approached the shift. In both cases, we contacted the upstreams in good time to ascertain their view on their projects’ readiness, then agreed funding to ensure they had the financial support they needed to land changes in support of Ubuntu, and then worked closely with them throughout the cycle to solve various performance and implementation issues we discovered along the way.

As it stands today, sudo-rs is the default sudo implementation on Ubuntu 25.10, and uutils’ coreutils has mostly replaced the GNU implementation, with a few exceptions, many of which will be resolved by releases in the coming weeks. These diversions back to the existing implementations demonstrate that stability and resilience are more important than “hype” in our approach: I expect us to have completed the migration during the next cycle, but not before the tools are ready.

Following the “Switch to Dracut” specification, Ubuntu Desktop 25.10 will use Dracut as its default initrd infrastructure (replacing initramfs-tools). Dracut will use systemd in the initrd and supports new features like Bluetooth and NVMe over Fabric (NVM-oF) support. Ubuntu Server installations will continue using initramfs-tools until remaining hooks are ported.

For each of these changes (coreutils, sudo-rs and dracut) the previous implementations will remain supported for now, with well-documented instructions on the reversion of each change for those who run into unavoidable issues - though we expect this to be a very small number of cases.

What’s Next? #

Well… more of the same! We intend to carry on with the increased cadence of written updates, so keep an eye out for those.

We have some exciting announcements to make over the coming weeks, including support for more modern micro-architectural variants (like amd64v3), better system-wide handling of revoked TLS certificates, updates on our Debcraft package for a more modern packaging experience and an effort to update many of our tools from “behind the scenes” using a combination of Rust and Go.

My final words are to thank all of those who have driven these efforts. I’ll omit the long list of names, but there have been countless examples of people stepping up substantially to deliver these efforts - without whom we’d have made a lot less progress.

Well done, and let’s make Resolute Raccoon an LTS to remember - for all the right reasons!

The Immutable Linux Paradox

Mon, 01 Sep 2025 00:00:00 +0000

Immutable Linux distributions have been around since the early 2000s, but adoption has significantly accelerated in the last five years. Mainstream operating systems (OSes) such as macOS, Android, ChromeOS and iOS have all embraced similar principles, reflecting a growing trend toward resilience, longevity, and maintainability as core ideals of OS development.

Ubuntu Core has been at the forefront of this movement for IoT, appliances and edge deployments, with work ongoing to release a “Core Desktop” experience. Other projects such as NixOS, Fedora Silverblue and Red Hat image mode are gaining adoption, alongside more specialised immutable distributions such as SteamOS and Talos.

This post explores how different Linux distributions achieve immutability, the trade-offs, and why you should give it a try!

What is an immutable Linux distribution? #

The key principle of an immutable OS is that the core system is unchangeable at runtime.

Every OS installation has at least one filesystem that stores system software, user software, and user data. Immutable OSes must cleanly separate “system” and “user” software and data, such that regular user interactions cannot compromise the integrity of the OS.

Immutable deployments are often separated into three layers:

Base OS - immutable core, updated only through controlled mechanisms
Applications - user applications, often delivered in containerised formats such as Snap, Flatpak, AppImage, cpak
User data - writable and persistent, independent of OS updates or rollbacks

Immutable systems use atomic, transactional updates meaning updates are applied as unitary, indivisible operations that either wholly succeed, or fail completely and trigger an automated roll-back to a previous known-good state.

Why immutability? #

The major benefit of an immutable OS is resilience.

Immutable OSes make it easier to reproduce systems with a given configuration, which is particularly useful in scale-out use-cases such as cloud or IoT.

Traditional package managers often maintain a database of installed packages, consisting of those included in the base OS, and those explicitly installed by the user, and their dependencies. The package manager doesn’t have a clear notion of which packages make up the “core system”, and which are “optional”.

This can cause “configuration drift”, which occurs over time - a package could be explicitly installed by a user, used for a while and then removed, but without removing its dependencies. This leaves the system in a different, and somewhat undefined, state than it was in prior to the package being installed.

Often the traditional notion of OS security is improved with immutable OS concepts too. In most implementations, the core OS files are mounted read-only such that users cannot make changes - which also raises the bar for malicious modifications. When combined with technologies such as secure boot and confinement, immutable OSes can dramatically reduce the attack surface of a machine.

Finally, convenience! Immutable OSes often include recovery or rollback features, which enable users to “undo” a bad system change, reverting to a previous known-good revision.

The immutability paradox #

In reality, no general-purpose operating system is fully immutable.

There is always persistent, user-writable storage - because without this there would be a huge limitation on usefulness! Similarly, how can a system be truly immutable, yet still support software updates?

The terms “immutable” and “stateless” are often conflated - when in reality neither are excellent terms for describing what has become widely known as “immutable OSes”. This was explored in some depth in this blog post which proposes terms such as “image based” and “fully managed”.

By definition, changes to configuration, the installation of applications and the use of temporary runtime storage are all violations of immutability, and thus immutability concepts must be applied in some sort of layering system.

Striking the balance between ’true’ immutability and user experience is one of the hardest challenges in immutable OS design. A system that is too rigid can be difficult to manage and use, appearing inflexible to end users.

A common pattern is to run an immutable desktop OS and use virtualisation or containerisation technologies (e.g. LXD, Podman, toolbx Distrobox) to create mutable environments in which to work on projects. This results in a very stable workstation that benefits from immutability, with the flexibility of a traditional mutable OS where it’s needed.

Approaches to immutability #

Different distributions solve the immutability challenge in different ways. In this section we’ll explore the four different approaches of ostree based distributions, bootc based distributions, NixOS and Ubuntu Core.

Fedora Silverblue / CoreOS / EndlessOS (`ostree`) #

Fedora Silverblue and Fedora CoreOS are also popular choices for those exploring immutable OSes. The two share a lot of underlying technology with Silverblue targeting desktop use cases, and CoreOS targeting server deployments.

Both are based on ostree, which provides:

Silverblue and CoreOS actually rely on rpm-ostree , a “hybrid image/package manager” which combines RPM packaging technology with ostree to manage deployments.

The update mechanism involves switching the filesystem to track a different remote “ref”, which is analogous to a git ref.

EndlessOS is based on Debian, but uses ostree to achieve immutability. EndlessOS is a desktop experience designed more for the “average user” and focuses on providing a reliable system that works well in low-bandwidth or offline situations.

Users often use Flatpak to install graphical user applications atop the immutable base, or a user-space package manager such as brew for other utilities.

ostree based distributions also support “package layering” which enables adding packages to the base system without fetching a whole new filesystem ref, but does require the system to be rebooted before the package is persistently available. The documentation notes that this approach is to be used “sparingly”, and that users should prefer using Flatpak or toolbx to access additional packages.

RHEL “Image Mode” (`bootc`) #

bootc based distributions use an alternate approach, packaging the base system into OCI containers (commonly referred to as Docker containers). Atomicity and transactionality are achieved by using container images to deliver the entire core system, and rebooting into a new revision.

RHEL Image Mode uses bootc. This technology capitalises on the success of OCI containers as a transport and delivery mechanism for software by packing an entire OS base image into a single container, including the kernel image.

The bootc project builds on ostree , but where ostree never delivered an opinionated “install mechanism”, bootc does. The contents of a bootc image is an ostree filesystem.

Installing new system packages generally means building a new base image, downloading that image and rebooting into it with a command such as bootc switch <image reference>.

Users often use Flatpak to install graphical user applications atop the immutable base, or a user-space package manager such as brew for other utilities.

NixOS #

The Nix project first appeared in 2003. NixOS is built on top of the Nix package manager, using it to manage both packages and system configuration.

NixOS defines the entire system through a declarative configuration, with changes applied via “generations” that can be rolled back. Changes to the system are applied by “rebuilding” the system configuration, which produces a new “generation”.

Nix packages, and therefore NixOS, eschews the traditional Unix FHS in favour of the Nix “store” and a collection of symlinks and wrappers managed by Nix. Only the Nix package manager can write to the store.

The Nix store also (mostly) enables the building and switching of generations without a reboot. Updates are atomic: new generations must build completely before they can be activated. The home-manager project extends these concepts to the user environment and dotfile management.

The impermanence project requires that every persistent directory is explicitly labelled, or else it’s deleted on every reboot, forcing the base OS to be rebuilt from the Nix store and system configuration - essentially “enforcing” core system immutability between reboots. This was inspired by blog posts “Erase Your Darlings” and “NixOS tmpfs as root”, which are worth a read, too!

Ubuntu Core #

Ubuntu Core achieves immutability by packaging every component (kernel, base system, applications) as Snaps.

Snap confinement enforces isolation, and snapd manages transactional updates and rollbacks. The system is designed for reliability, fleet management, and modular upgrades, making it well-suited for IoT and soon, desktop use.

The key components of an Ubuntu Core deployment are:

Gadget snap: provides boot assets, including board specific binaries and data (bootloader, device tree, etc.)
Kernel snap: kernel image and associated modules, along with initial ramdisk for system initialisation
Base snap: execution environment in which applications run - includes “base” Ubuntu LTS packages
System snaps: packages critical to system function such as Network-Manager, bluez, pulseaudio, etc.
Application snaps: define the functionality of the system, confined to a sandbox
Snapd: manages updates, rollbacks and snapshotting/restoring of user data

In a Core Desktop installation, the desktop environment (GNOME, Plasma, etc.), display manager, login manager would all be delivered as “system snaps”.

Snap confinement ensures packages cannot incorrectly interact with the underlying system or user data without explicit approval. In an Ubuntu Core deployment, this notion is extended to every component of the OS, offering a straightforward yet powerful way to manage risk for each system component.

Summary #

Immutable Linux distributions approach the immutability paradox differently. We explored four different approaches here, and you can learn more about other approaches taken by the likes of SUSE MicroOS (filesystem based immutability) and Vanilla OS (uses ABRoot) in this excellent blog post.

Ubuntu Core focuses on transactional packaging and a clean separation of system & user data. bootc-based systems take a full image-based approach, while NixOS offers extreme flexibility through declarative configuration, but at the cost of complexity.

If you’ve yet to try an immutable Linux distribution I’d recommend giving it a go. Whether you prioritise simplicity, security or declarative control there’s almost certainly an immutable Linux distribution that fits your needs.

Crafting Your Software

Mon, 21 Jul 2025 00:00:00 +0000

Packaging software is notoriously tricky. Every language, framework, and build system has its quirks, and the variety of artifact types — from Debian packages to OCI images and cloud images — only adds to the complexity.

Over the past decade, Canonical has been refining a family of tools called “crafts” to tame this complexity and make building, testing, and releasing software across ecosystems much simpler.

The journey began on 23rd June 2015 when the first commit was made to Snapcraft, the tool used to build Snap packages. For years, Snapcraft was the only craft in our portfolio, but in the last five years, we’ve generalized much of what we learned about building, testing, and releasing software into a number of “crafts” for building different artifact types.

Last month, I outlined Canonical’s plan to build debcraft as a next-generation way to build Debian packages. In this post I’ll talk about what exactly makes a craft, and why you should bother learning to use them.

Software build lifecycle #

At the heart of all our crafts is craft-parts, which according to the documentation “provides a mechanism to obtain data from different sources, process it in various ways, and prepare a filesystem sub-tree suitable for packaging”.

Put simply, craft-parts gives developers consistent tools to fetch, build, and prepare software from any ecosystem for packaging into various formats.

Lifecycle stages #

Every part has a minimum of four lifecycle stages:

PULL: source code or binary artifacts, along with dependencies are pulled from various sources
BUILD: software is built automatically by a plugin, or a set of custom steps defined by the developer
STAGE: select outputs from the BUILD phase are copied to a unified staging area for all parts
PRIME: files from the staging area are copied to the priming area for use in the final artifact.

The STAGE and PRIME steps are similar, except that PRIME only happens after all parts of the build are staged. Additionally, STAGE provides the opportunity for parts to build/supply dependencies for other parts, but that might not be required in the final artifact.

Lifecycle in the CLI #

The lifecycle stages aren’t just in the build recipe, they’re also first-class citizens in each craft’s CLI, thanks to the craft-cli library. This ensures a consistent command-line experience across all craft tools.

Take the following examples:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Run the full process including PULL, BUILD, STAGE, PRIME and then pack the final artifact
snapcraft pack
charmcraft pack
rockcraft pack

# Run the process up to the end of the STAGE step
rockcraft stage

# Run the process up to the PRIME step
charmcraft prime

This design feature supports a smoother iterative development and debugging workflow for building and testing software artifacts.

Part definition #

The parts of a build vary in complexity - some require two-three trivial lines, others require detailed specification of dependencies, build flags, environment variables and steps. The best way to understand the flexibility of this system is by looking at some examples.

First, consider this (annotated) example from my icloudpd snap:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


icloudpd:
 # Use the 'python' plugin to build the
 # software. This takes care of identifying
 # Python package dependencies, building the wheel
 # and ensuring the project's dependencies are staged
 # appropriately.
 plugin: python
 # Fetch the project from Github, using the tag the matches
 # the version of the project.
 source: https://github.com/icloud-photos-downloader/icloud_photos_downloader
 source-tag: v$SNAPCRAFT_PROJECT_VERSION
 source-type: git

This spec is everything required to fetch, build and stage the important bits required to run the software - in this case a Python wheel and its dependencies.

Some projects might require more set up, perhaps an additional package is required or a specific version of a dependency is needed. Let’s take a look at a slightly more complex example taken from my zinc-k8s-operator project:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


kube-log-runner:
 # Use the 'go' plugin to build the software.
 plugin: go
 # Fetch the source code from Git at the 'v0.17.0' tag.
 source: https://github.com/kubernetes/release
 source-type: git
 source-tag: v0.17.8
 # Change to the specified sub-directory for the build.
 source-subdir: images/build/go-runner
 # Install the following snaps in the build environment.
 build-snaps:
 - go/1.20/stable
 # Set the following environment variables in the build
 # environment.
 build-environment:
 - CGO_ENABLED: 0
 - GOOS: linux

This instructs rockcraft to fetch a Git repository at a particular tag, change into the sub-directory images/build/go-runner, then build the software using the go plugin. It also specifies that the build required the go snap from the 1.20/stable track, and sets some environment variables. That’s a lot of result for not much YAML. The end result of this is a single binary that’s “staged” and ready to be placed (in this case) into a Rock (Canonical’s name for OCI images).

And the best part: this exact definition can be used in a rockcraft.yaml when building a Rock, a snapcraft.yaml when building a Snap, a charmcraft.yaml when building a Charm, etc.

The plugin system is extensive: at the time of writing there are 22 supported plugins, including go, maven, uv, meson and more. If your build system of choice isn’t supported you can specify manual steps, giving you as much flexibility as you need:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


wasi-sdk:
 # There is no appropriate plugin for this part, so set
 # it to 'nil' and we'll specify our own build process
 # using 'override-build'.
 plugin: nil
 # In this recipe, a previous part named 'clang' is
 # required to build before attempting to build this
 # part.
 after:
 - clang
 # Specify any `apt` packages required in the build
 # environment.
 build-packages:
 - wget
 # Set some environment variables for the build
 # environment.
 build-environment:
 - WASI_BRANCH: "15"
 - WASI_RELEASE: "15.0"
 # Define how to pull the software manually.
 override-pull: |
 ROOT=https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-$WASI_BRANCH
 wget $ROOT/wasi-sysroot-$WASI_RELEASE.tar.gz
 wget $ROOT/libclang_rt.builtins-wasm32-wasi-$WASI_RELEASE.tar.gz
 # Define how to 'build' the software manually
 override-build: |
 craftctl default
 tar -C $CRAFT_STAGE -xf wasi-sysroot-$WASI_RELEASE.tar.gz
 tar -C $CRAFT_STAGE/usr/lib/clang/* -xf libclang_rt.builtins-wasm32-wasi-$WASI_RELEASE.tar.gz
 # Don't prime anything for inclusion in the
 # final artifact; this part is only used for
 # another part's build process.
 override-prime: ''

Here, multiple stages of the lifecycle are overridden using override-build, override-pull and override-stage, and we see craftctl default for the first time, which instructs snapcraft to do whatever it would have done prior being overridden, but allows the developer to provide additional steps either before or after the default actions.

Isolated build environments #

Even once a recipe for building software is defined, preparing machines to build software can be painful. Different major versions of the same OS might have varying package availability, your team might run completely different operating systems, and you might have limited image availability in your CI environment.

The crafts solve this with build “backends”. Currently the crafts can use LXD or Multipass to create isolated build environments, which makes it work nicely on Linux, macOS and Windows. This functionality is handled automatically by the crafts through the craft-providers library. The craft-providers library provides uniform interfaces for creating build environments, configuring base images and executing builds.

This means if you can run snapcraft pack on your machine, your teammates can also run the same command without worrying about installing the right dependencies or polluting their machines with software and temporary files that might result from the build.

One of my favourite features of this setup is the ability to drop into a shell inside the build environment automatically on a few different conditions:

1
2
3
4
5
6


# Drop into a shell if any part of the build fails.
snapcraft pack --debug
# Drop into a shell after the build stage.
rockcraft build --shell-after
# Drop to a shell in lieu of the prime stage.
snapcraft prime --shell

This makes troubleshooting a failing build much simpler, while allowing the developer to maintain a clean separation between the build environment and their local machine. Should the build environment ever become polluted, or otherwise difficult to work with, you can always start from a clean slate with snapcraft|rockcraft|charmcraft clean. Each build machine is constructed using a cached build-base, which contains all the baseline packages required by the craft - so recreating the build environment for a specific package only requires that base to be cloned and augmented with project specific concerns - making the process faster.

Saving space #

When packaging any kind of software, a common concern is the size of the artifact. This might be because you’re building an OCI-image that is pulled thousands of times a day as part of a major SaaS deployment, or maybe it’s a Snap for an embedded device running Ubuntu Core with a limited flash. In the container world, “distroless” became a popular way to solve this problem - essentially popularising the practice of shipping the barest minimum in a container image, eschewing much of the traditional Unix FHS.

The parts mechanism has provided a way of “filtering” what is staged or primed into a final artifact from the start, which already gave developers autonomy to choose exactly what went into their builds.

In addition to this, Canonical built “chisel”, which extends the distroless concept beyond containers to any kind of artifact. With chisel, developers can slice out just the binaries, libraries, and configuration files they need from the Ubuntu Archive, enabling ultra-small packages without losing the robustness of Ubuntu’s ecosystem.

We later launched Chiseled JRE containers, and there are numerous other Rocks that utilise chisel to provide a balance between shipping tiny container images, while benefiting from the huge selection and quality of software in the Ubuntu Archive.

Because the crafts are all built on a common platform, they now all have the ability to use “slices” from chisel-releases, which enables a greater range of use-cases where artifact size is a primary concern. Slices are community maintained, and specified in simple to understand YAML files. You can see the list of available slices for the most recent Ubuntu release (25.04 Plucky Puffin) on GitHub, and further documentation on slices and how they’re used in the Chisel docs.

Multi-architecture builds #

Ubuntu supports six major architectures at the time of writing (amd64, arm64, armhf, ppc64le, s390x, riscv64), and all of our crafts have first-class support for each of them. This functionality is provided primarily by the craft-platforms library, and supported by the craft-grammar library, which enables more complex definitions where builds may have different steps or requirements for different architectures.

At a high-level, each artifact defines which architectures or platforms it is built for, and which it is built on. These are often, but not always, the same. For example:

1
2


platforms:
 amd64:

This is shorthand for “build the project on amd64 for amd64”, but in a different example taken from a charmcraft.yaml

1
2
3
4


platforms:
 all:
 build-on: [amd64]
 build-for: [all]

In this case the software is built on amd64, but can run on any of the supported architectures - this can happen with all-Python wheels, bash scripts and other interpreted languages which don’t link platform-specific libraries.

In some build processes, the process or dependencies might differ per-architecture, which is where craft-grammar comes in, enabling expressions such as (taken from GitHub):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


fit-image:
 # ...
 build-packages:
 # ...
 - wget
 - libjson-c-dev:${CRAFT_ARCH_BUILD_FOR}
 - libcryptsetup-dev:${CRAFT_ARCH_BUILD_FOR}
 # Only use the following build packages when building for armhf
 - to armhf:
 - binutils-arm-linux-gnueabi
 - gcc-arm-linux-gnueabihf
 - pkgconf:armhf
 # When building for arm64, use a different set
 - to arm64:
 # Dependencies for building *for* arm64 *on* amd64!
 - on amd64:
 - gcc-aarch64-linux-gnu
 - pkgconf:arm64
 - on arm64:
 - gcc

Being able to define how to build on different architectures is only half of the battle, though. It’s one thing to define how to build software on an s390x machine but few developers have mainframes handy to actually run the build! This is where the crafts’ remote-build capability comes in. The remote-build command sends builds to Canonical’s build farm, which has native support for all of Ubuntu’s supported architectures. This is built into all of our crafts, and is triggered with snapcraft remote-build, rockcraft remote-build, etc.

Remote builds are a lifeline for publishers and communities who need to reach a larger audience, but can’t necessarily get their own build farm together. One example of this is Snapcrafters, a community-driven organisation that packages popular software as Snaps, who use remote-build to drive multi-architecture builds from GitHub Actions as part of their publishing workflow (as seen here and here for example).

Unified testing framework #

Testing is often the missing piece in build tools: developers are forced to rely on separate CI systems or ad-hoc scripts to verify their artifacts. To close this gap, we’re introducing a unified test sub-command in the crafts.

We recently added the test sub-command to our crafts as an experimental (for now!) feature. Under the hood, craft test will introduce a new lifecycle stage (TEST). The enables packagers of any artifact type to specify how that artifact should be tested using a common framework across artifact types.

Craft’s testing capability is powered by spread, a convenient full-system task distribution system. Spread was built to simplify the massive number of integration tests run for the snapd project. It enables developers to specify tests in a simple language, and distribute them concurrently to any infrastructure they have available.

This enables a developer to define tests and test infrastructure, and make it trivial to run the same tests locally, or remotely on cloud infrastructure. This can really speed up the development process - preventing developers from needing to wait on CI runners to spin up and test their code while iterating, they can run the very same integration tests locally using craft test.

There are lots of fine details to spread, and the team is working on artifact-specific abstractions for the crafts that will make testing delightful. Imagine maintaining the Snap for a GUI application, and being able to enact the following workflow:

1
2
3
4
5
6
7
8
9


# Pull the repository
git clone https://github.com/some-gui-app/snap && cd snap
# Make some changes, perhaps fix a bug
vim snap/snapcraft.yaml
# Build the snap, and run the integration tests.
# These tests might include spinning up a headless
# graphical VM, which actually installs and runs
# the snap, and interacts with it
snapcraft test

By integrating a common testing tool into the build tooling, the Starcraft team will be able to curate unique testing experiences for each kind of artifact. A snap might need a headless graphical VM, where an OCI-image simply requires a container runtime, but the spread underpinnings allow a common test-definition language for each.

There are a couple of examples of this in the wild already:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# Install charmcraft
sudo snap install --classic charmcraft
# Clone the repo
git clone https://github.com/jnsgruk/zinc-k8s-operator
cd zinc-k8s-operator
# List the available tests
charmcraft test --list lxd:
# Run the integration testing suite, spinning up
# a small VM, inside which is a full Kubernetes
# instance, with a Juju controller bootstrapped.
# From here the charm will be deployed and tested to
# ensure it's integrations with the observability
# stack and ingress charms are functioning correctly.
charmcraft test -v lxd:ubuntu-24.04:tests/spread/observability-relations:juju_3_6

The test above is powered by this spread.yaml, and this test definition. With a little bit of work, it’s also possible to integrate spread with GitHub matrix actions, giving you one GitHub job per spread test - as seen here.

You can see a similar example in our PostgreSQL Snap test suite, and we’ll be adding more and more of this kind of test across our Rock, Snap, Charm, Image and Deb portfolio.

There is work to do, but I’m really excited about bringing a common testing framework to the crafts which should make the testing of all kinds of artifacts more consistent and easier to integrate across teams and systems.

Crafting the crafts #

As the portfolio expanded from snapcraft, to charmcraft, to rockcraft and is now expanding further to debcraft and imagecraft it was clear that we’d need a way to make it easy to build crafts for different artifacts, while being rigorous about consistency across the tools. A couple of years ago, the team built the craft-application base library, which now forms the foundation of all our crafts.

The craft-application library combines many of the existing libraries that were in use across the crafts (listed below), providing a consistent base upon which artifact-specific logic can be built. The allows craft developers to spend less time implementing CLI details, parts lifecycles and store interactions, and more time on curating a great experience for the maintainers of their artifact type.

For the curious, craft-application builds upon the following libraries:

craft-archives: manages interactions with apt package repositories
craft-cli: CLI client builder that follows the Canonical’s CLI guidelines
craft-parts: obtain, process, and organize data sources into deployment-ready filesystems.
craft-grammar: advanced description grammar for parts
craft-providers: interface for instantiating and executing builds for a variety of target environments
craft-platforms: manage target platforms and architectures for craft applications
craft-store: manage interactions with Canonical’s software stores
craft-artifacts: pack artifacts for craft applications

Examples and docs #

Before I leave you, I wanted to reference a few *craft.yaml examples, and link to the documentation for each of the crafts, where you’ll find the canonical (little c!) truth on each tool.

You can find documentation for the crafts below:

And some example recipes:

Snap: icloudpd - snapcraft.yaml
Snap: parca-agent - snapcraft.yaml
Snap: signal-desktop - snapcraft.yaml
Charm: ubuntu-manpages-operator - charmcraft.yaml
Rock: grafana - rockcraft.yaml
Rock: temporal-server - rockcraft.yaml

Summary #

The craft ecosystem provides developers with a rigorous, consistent and pleasant experience for building many kinds of artifacts. At the moment, we support Snaps, Rocks and Charms but we’re actively developing crafts for Debian packages, cloud images and more.The basic build process, parts ecosystem and foundations of the crafts are “battle tested” at this point, and I’m excited to see how the experimental craft test commands shape up across the crafts.

One of the killer features for the crafts is the ability to reuse part definitions across different artifacts - which makes the pay off for learning the parts language very high - it’s a skill you’ll be able to use to build Snaps, Rocks, Charms, VM Images and soon Debs!

If I look at ecosystems like Debian, where tooling like autopkgtest is the standard, I think debcraft test will offer an intuitive entrypoint and encourage more testing, and the same is true of Snaps, both graphical and command-line.

That’s all for now!

Introducing Debcrafters

Mon, 30 Jun 2025 00:00:00 +0000

Earlier this year, Canonical’s Ubuntu Engineering organisation gained a new team, seeded with some of our most prolific contributors to Ubuntu. Debcrafters is a new team dedicated to the maintenance of the Ubuntu Archive.

The team’s primary goal is to maintain the health of the Ubuntu Archive, but its unique construction aims to attract a broad range of Linux distribution expertise; contributors to distributions like Debian, Arch Linux, NixOS and others are encouraged to join the team, and will even get paid to contribute one day per week to those projects to foster learning and idea sharing

Bootstrapping the team #

The Debcrafters team is a global team. We have a squad in the Americas, a squad in EMEA and will have a squad in APAC. At present, we’ve staffed the AMER and EMEA teams with existing Canonical employees from our Foundations, Desktop, Server and Public Cloud teams. Each team currently has a manager, and four engineers.

The team comprises Debian Developers, Stable Release Updates (SRU) team members and archive administrators, and began working together for the first time at our recent Engineering Sprint in Frankfurt held in early May 2025.

Mission #

The Debcrafters’ primary mission is to maintain the health of the Ubuntu Archive.

This team will take the lead on syncing & merging packages from Debian, reviewing proposed migration issues, upstreaming Ubuntu deltas, and take ownership of major transitions such as upgrades to glibc and past examples such as the t64 and python3 transitions.

They’ll manage the scheduling, triggering and reporting on archive test rebuilds which we conduct when making major changes to critical packages. We did this when we enabled frame pointers by default, and when we switched coreutils to the uutils implementation in Ubuntu 25.10.

They’ll be responsible for the evolution and maintenance of the autopkgtest infrastructure for Ubuntu, as well as taking an instrumental role in introducing more distro-scale integration tests.

They’ll work on improving the reporting and dashboarding of the Ubuntu Archive, its contributors and status, as well as taking a broader interest in shaping the tools we use to build and shape Ubuntu.

What sets this team apart from the likes of Desktop, Server and Foundations is the range of packages they will work on. Members of the Debcrafters team will move thousands of packages every cycle - many of which they will not be intimately familiar with, but will use their growing distro maintenance and packaging skills to perform maintenance where there is no other clear or present owner.

Tools & processes #

One of the key goals in my first post was to modernise the contribution experience for Ubuntu Developers by focusing on tools and processes.

The Debian project recently adopted tag2upload, which allows Debian Developers to use git-debpush to push a signed git tag when uploading packages. While we’re not following that exact path, we share many of the same goals and intentions.

For some time Ubuntu Developers have been able to use git-ubuntu as part of their development workflow, which aims to provide “unified git-based workflows for the development of Ubuntu source packages”. This project brought us closer to our desired experience, but still needs work to achieve our complete vision. I’d like to put more emphasis on the experience we provide for testing packages, as well as signing, uploading and releasing packages.

In the coming weeks our Starcraft team (responsible for Snapcraft, Rockcraft, Charmcraft) will begin prototyping debcraft, which will (in time) become the de facto method for creating, testing and uploading packages to the Ubuntu archive.

The first prototype of debcraft will focus on unifying the current workflow adopted by most Ubuntu Developers at Canonical. It will wrap existing tools (such as git-ubuntu, lintian, autopkgtest) to provide familiar, streamlined commands such as debcraft pack, debcraft lint and debcraft test. Uploading packages, and a more native “craft” experience for constructing packages will come later.

Details will make their way into the new Ubuntu Project Docs throughout the course of the 25.10 Questing Quokka cycle, including the newly renovated “Ubuntu Packaging Guide”, which will aim to provide a “one ring to rule them all” approach to documenting how to package software for Ubuntu.

Attracting contributors #

While the team has been seeded with seasoned Ubuntu contributors, one of the primary goals of the team is to grow the contributor base across generations.

One of the sub-teams is currently leading the roll out of a new contributor journey that will soon be publicly available. This process lays out the journey from complete beginner to “Core Dev”, stopping off at “Package Maintainer”, “Package Set Maintainer”, “MOTU”, etc. along the way. The process also aims to help candidates prepare for Developer Membership Board interviews.

Whether you’re a junior engineer just graduating from University, or you’re a seasoned Linux contributor elsewhere in the Linux ecosystem, the Debcrafters team is an excellent place to learn software packaging skills and contribute to the world’s most deployed Linux distribution.

Contribution beyond Ubuntu #

The Debcrafters’ primary commitment is to Ubuntu, but we recognise the enormous value in collaborating with other distributions. Many of the hard lessons I’ve personally learned resulted from contributing to NixOS and building Snaps. Packaging is a complex and ever-changing discipline, and other distributions are facing many of the complex problems we are - often with different or novel approaches to solving them.

In recognition of this, we’re actively seeking maintainers from other distributions - be that Debian, Arch, NixOS, Guix, Fedora, Universal Blue or any other - packaging and distribution engineering skills are often common across distributions, and we believe that Ubuntu can benefit from broader perspectives, while contributing back to the wider ecosystem of distributions in the process.

The Debcrafters must spend the majority of their work time on Ubuntu, but they will be encouraged to spend a day per week contributing to other distributions to gain understanding, and bring fresh perspectives to Ubuntu (and the reverse, hopefully!). This will be structured as a literal day per week, agreed with the team management - for example “I work on NixOS on Tuesdays”.

Summary #

Canonical has launched a new team, the Debcrafters, who are dedicated to maintaining the very core of Ubuntu: the archive. This team has a global footprint, and deep expertise in software packaging drawn from across the Linux ecosystem. They’ll lead transitions, improve tooling improvements and strengthen our distribution testing infrastructure.

Whether you’re an experienced Debian Developer, a maintainer from another Linux distribution or a new engineer starting your career in open source, Debcrafters offers a unique opportunity to learn, grow, and contribute to the world’s most widely deployed Linux distribution.

Supercharging Ubuntu Releases: Monthly Snapshots & Automation

Thu, 29 May 2025 00:00:00 +0000

Introduction #

Ubuntu has shipped on a predictable, six-month cadence for two decades. Twenty years ago, the idea of releasing an entire distribution every six months was considered forward looking, bold and even difficult. Things have changed since then: software engineering has evolved as a practice, and the advent of both rolling-release distributions like Arch Linux, and more recently image-based immutable distributions such as Universal Blue have meant that other projects with similar goals have adopted vastly different release models with some desirable properties.

My goal over the coming months is to build a release process that takes advantage of modern release engineering practices, while retaining the resilience and stability of our six-monthly releases. We’ll introduce significantly more automated testing, and ensure that the release process is transparent, repeatable and executable in a much shorter and well-known timeframe with little to no human intervention.

This journey will also create space for better system-wide testing, earlier detection of regressions, and a more productive collaboration with our community.

Monthly Snapshot Releases #

Starting in May 2025, we’re introducing monthly snapshot releases for Ubuntu.

Ubuntu is not “moving to monthly releases” or adopting a rolling release model; we’re committed to our six-monthly releases with a Long Term Support (LTS) release every two years. That doesn’t mean that our release process should be exempt from the same scrutiny that the rest of our engineering processes are subject to.

Today the Ubuntu Release process is the product of twenty years of evolution: it safeguards Ubuntu releases with a wealth of checks and balances, but is a largely manual process requiring significant human involvement.

The Ubuntu Release Team is a crowd of seasoned Ubuntu veterans who have been steadily releasing Ubuntu for many years. Many of this team are community members, some are or have been employed by Canonical in the past. More recently we have established the Canonical Ubuntu Release Management Team - a relatively new team at Canonical who’ll be collaborating with the Ubuntu Release Team to develop the new process.

To aid the Canonical team in their understanding of the existing processes, and the immovable requirements that sit beneath it, we’re introducing monthly snapshot releases for Ubuntu. These will not be fully-fledged releases of Ubuntu, but rather curated, testable milestones from our development stream. For the 25.10 (Questing Quokka) cycle, you can expect the following release schedule:

May 29, 2025: Questing Quokka - Snapshot 1
June 26, 2025: Questing Quokka - Snapshot 2
July 31, 2025: Questing Quokka - Snapshot 3
August 28, 2025: Questing Quokka - Snapshot 4
September 18, 2025: Questing Quokka - Beta
October 9, 2025: Questing Quokka - Final Release

This doesn’t mean you’ll start seeing Ubuntu versions off the six-month cadence. There will be no Ubuntu 25.07 or 25.08, etc. The monthly snapshots are exactly that: a snapshot of the development of Ubuntu 25.10. Snapshots are not meant for production use, but will help the release team move away from deep institutional knowledge, and toward clean well-documented automated workflows that are transparent, repeatable and testable.

With our current model, failure modes are not detected until they’re urgent and blocking an imminent release. The team conducts rigorous retrospectives on each release, but in my opinion it’s hard to meaningfully evolve such a process when it’s only exercised every six months. The monthly snapshots will create opportunities for us to test, understand and improve the process.

Embracing Automation #

One of the most valuable outcomes of this journey will be the opportunity to automate more of the process, freeing up time for the team to focus on more strategic tasks. Releasing a distribution is a complex process requiring coordination across architectures, images, mirrors, websites, testing infrastructure and even partner agreements. This also makes it hard to place a traditional CI tool at the heart of the process. As much as I like Github Actions, I think we’d quickly get lost trying to release Ubuntu with such a system, notwithstanding the fact that we’d lose control of the underlying infrastructure that releases Ubuntu.

I’ve been exploring the world of Durable Execution, which according to restate.dev is:

At Canonical, we’ve adopted Temporal in a few of our products and in many of our business processes. Temporal is a durable execution product that enables developers to solve complex distributed problems, but without being deep distributed systems experts. It’s a framework for composing tasks into workflows, with first-class primitives for dealing with failures, retries, exponential back-off and other concepts that enable the build of long-running complex workflows.

Having spent some time with Temporal myself, and watched other teams adopt it, I think it’s a great fit for engineering our next-generation release process. I want our engineers to focus on the logic of the release process, not the infrastructure behind it, and Temporal should enable them to do just that. The Temporal homepage sums it up nicely:

Temporal workflows and activities can be written in many languages - and particularly in Python and Go. My expectation is that Go will prove to be an excellent fit for our process: it’s a fast and productive language that specialises in concurrency and asynchronous network operations, and has a powerful standard library containing much of the functionality we’ll need to build our new release process.

To take an overly simplistic view of how I expect this to go: we’ll take our existing release checklist, write a Go function for each step with some tests, and compose them together into one or more Temporal workflows that represent the full release process. This will take time, but this approach will enable us to incrementally demonstrate progress toward a fully-automated process over the coming cycles.

By making this move, not only will we make the process quicker, but also more observable, testable, reliable and easier to understand for everyone, not just the release team.

Improving Test Coverage #

One area I’d like to improve as a side-effect of this work is more full-system integration testing. Packages in the Ubuntu archive generally enjoy good coverage through a suite of autopkgtest tests, and there are numerous other places where integration tests are run on Ubuntu. With our traditional six-monthly cadence, full end-to-end testing of ISOs and the installer typically ramps up close to release time when changes are fewer (and riskier) and time is short.

With the introduction of monthly snapshots, we can integrate installer testing, full-disk encryption testing, graphical application testing and more as a regular, automated part of the release pipeline - not just as part of the development pipeline of each individual package. This means we should catch regressions earlier and surface more edge cases to be resolved before release.

One of the most important parts of increasing our testing culture is to make it clear where and how to contribute tests to Ubuntu. The easier we make it to write and contribute tests, the more tests we’re likely to add to the suite. We’re doing some work on this in parallel which will likely turn into a blog post of its own in the coming months.

In our current process, we have a heroic group of volunteers who kindly spend hours on our behalf testing the various flavours - exercising all the possible install paths and validating that what is about to be published is fit for purpose. I’d like to ensure that our volunteers’ time is spent as productively and rewardingly as possible, and I think we can automate much of this testing and allow them to focus on the more complex and nuanced aspects of each release, and raise the quality of Ubuntu across all the flavours.

What’s Next? #

We’re starting by modeling the current release process as it is. Once we’ve validated our assumptions about the current process, we’ll layer in improvements by reducing manual gates, parallelising independent steps, introducing more testing, and exercising the process each month to test (and measure) any improvements we’ve made.

My ultimate goal is a release system that’s incredibly “boring”: transparent, predictable, observable, and easy to reason about (even when things go wrong).

The new, fully-automated process will likely take several months to complete. When we think we’re done, we’ll do a release that runs both processes in parallel to ensure we get the outcome we expect before finally sunsetting the old process.

We’ll be building this work in the open (and hiring!) so if you’ve used Temporal in similar contexts, or are curious about contributing to this effort, we’d love to hear from you.

Until next time!

Adopting sudo-rs By Default in Ubuntu 25.10

Tue, 06 May 2025 00:00:00 +0000

Introduction #

Following on from Carefully But Purposefully Oxidising Ubuntu, Ubuntu will be the first major Linux distribution to adopt sudo-rs as the default implementation of sudo, in partnership with the Trifecta Tech Foundation

The change will be effective from the release of Ubuntu 25.10. You can see the Trifecta Tech Foundation’s announcement here.

What is `sudo-rs`? #

sudo-rs is a reimplementation of the traditional sudo tool, written in Rust. It’s being developed by the Trifecta Tech Foundation (TTF), a nonprofit focused on building secure, open source infrastructure components. The project is part of the Trifecta Tech Foundation’s Privilege Boundary initiative, which aims to handle privilege escalation with memory-safe alternatives.

The sudo command has long served as the defacto means of privilege escalation on Linux. As described in the original post, Rust provides strong guarantees against certain classes of memory-safety issues, which is pivotal for components at the privilege boundary.

The sudo-rs team is collaborating with Todd Miller, who’s maintained the original sudo for over thirty years. sudo-rs should not be considered a fork in the road, but rather a handshake across generations of secure systems. Throughout the development of sudo-rs, the TTF team have also made contributions to enhance the original sudo implementation.

The sudo-rs project is designed to be a drop in replacement for the original tool. For the vast majority of users, the upgrade should be completely transparent to their workflow. That said, sudo-rs is a not a “blind” reimplementation. The developers are taking a “less is more” approach. This means that some features of the original sudo may not be reimplemented if they serve only niche, or more recently considered “outdated” practices.

Erik Jonkers, Chair of the Trifecta Tech Foundation explains:

Sponsoring Mainstream Adoption #

Leading the mainstream adoption of a replacement to such a universally understood tool comes with responsibility. Before committing to ship sudo-rs in Ubuntu 26.04 LTS, we’ll test the transition in Ubuntu 25.10. We’re also sponsoring the development of some specific items, which has manifested as Milestone 5 in the upstream project:

Coarse-grained shell escape prevention (NOEXEC) on Linux (See PR #1073)
The ability to control AppArmor profiles (First PR #1067)
A sudoedit implementation
Support for Linux Kernels older than version 5.9

The final item may seem out of place, but because Ubuntu 20.04 LTS is still in support, without this work there could be situations where sudo fails to function if, for example, a 26.04 LTS OCI container was run on a 20.04 LTS host!

The team have also already begun work on ensuring that the test-suite is as compatible as possible with Ubuntu, to ensure any issues are caught early.

This isn’t just about shipping a new binary. It’s about setting a direction. We’re not abandoning C, or even rewriting all the utilities ourselves, but by choosing to replace one of the most security-critical tools in the system with a memory-safe alternative, we’re making a statement: resilience and sustainability are not optional in the future of open infrastructure.

Progress on `coreutils` #

Since the initial announcement, we’ve been working hard to more clearly define a plan for the migration to uutils coreutils in 25.10 and beyond. Similarly to our engagement with the Trifecta Tech Foundation, we’re also sponsoring the uutils project to ensure that some key gaps are closed before we ship 25.10. The sponsorship will primarily cover the development of SELinux support for common commands such as mv, ls, cp, etc.

The first step toward developing SELinux support was to add support for automated testing in Github Actions, since then the maintainers have begun work on the actual implementation.

The other feature we’re sponsoring is internationalisation support. At present, some of the utility implementations (such as sort) have an incomplete understanding of locales, and therefore may yield unexpected results. We expect that these two features should land in time for us to ship in 25.10, and we’ll continue to work with the uutils project throughout the 26.04 LTS cycle to close any remaining gaps we identify in the interim release.

One of the major concerns outlined in Julian’s post is about binary size. We’ve got a few tricks we can play here to get the size down, and there is already some conversation started upstream in Debian on how that might be achieved. There are also security implications, such as AppArmor’s lack of support for multi-call binaries. We’re currently working with the respective upstreams to discuss addressing this systematically, through in the interim we may need to build small wrapper binaries to enable compatibility with existing AppArmor profiles from the start.

Migration Mechanics #

Julian Klode posted recently on the Ubuntu Discourse outlining the packaging plan that will enable us both to migrate transparently to uutils coreutils, but also provide a convenient means for users to opt-out and switch back to GNU coreutils if they wish, or if they identify a gap in the new implementation. I expect this will be rare, but we want to make sure it’s as easy as possible to revert, and will be documenting this in detail before release.

Replacing coreutils isn’t as simple as swapping binaries. As an Essential package, its replacement must work immediately upon unpacking without relying on maintainer scripts, and without conflicting files across packages. To solve this, we’re introducing new coreutils-from-uutils and coreutils-from-gnu packages, as well as coreutils-from itself. For all the gory details, see the Discourse post!

The packaging work required to switch to sudo-rs is somewhat less complicated than with coreutils. The package is already available in Ubuntu (which you can still test on Ubuntu 24.04, 24.10 and 25.04 with oxidizr!), but unlike coreutils, sudo is not an Essential package, so we’ll be able to make use of the Debian alternatives system for the transition.

Summary #

Things are progressing nicely. We’ve established strong, productive relationships and are sponsoring work upstream to make these transitions viable.

We’ve got a strategy for migrating the default implementation of coreutils and sudo in Ubuntu 25.10 which will enable a seamless revert in cases where that is desired. While sudo-rs will be the default in 25.10, the original sudo will remain available for users who need it, and we’ll be gathering feedback to ensure a smooth transition before the 26.04 LTS.

Additionally, we’ve begun investigating the feasibility of providing SequoiaPGP and using it in APT instead of GnuPG. SequoiaPGP is a new OpenPGP library with a focus on safety and correctness, written in Rust. The GnuPG maintainers have recently forked the OpenPGP standard and are no longer compliant with it. Sequoia provides a modern alternative to GnuPG with strict behavior, and is already used in various other systems. More details to follow!

Stay tuned!

Revitalising Ubuntu Project Documentation

Tue, 01 Apr 2025 00:00:00 +0000

Introduction #

Back in February I wrote some thoughts on Ubuntu’s documentation and its role within the community. For a mostly-online software community, documentation is one of our most critical forms of communication.

In the last two years there has been lots of focus on the technical aspects of our documentation (how-to guides, tutorials, etc.), but I’d like to focus more on what I’m calling the “Ubuntu Project Documentation” over the coming months.

Documentation isn’t only about technical how-to guides and tutorials, nor is it only about troubleshooting or satisfying particular use-cases. Our documentation can set the tone for the project, give a means for the community to state an intent, and guide both current and future contributors in their daily work.

Ubuntu has a lot of documentation, most of which has grown organically over the last 20 years, but it’s not always easy to find or understand. Our documentation should illuminate and inspire a path to contribution. It should provide direction and clarity on complex issues, reference on technology and past decisions, and precision in the execution of process.

Our project documentation should detail what makes Ubuntu happen. How are decisions made? What are the teams contributing to Ubuntu? How are those teams appointed? What are their responsibilities? If you’re on the Main Inclusion Review (MIR) team and you’re assigned a package to review, what steps should you take? How does package sponsorship work, and who should you contact if you’re stuck? How are the Access Control Lists (ACLs) updated for packages and package sets, and who can make those changes? What does the journey look like from first time package bug-fixer to Ubuntu Core Developer?

These are all examples of questions that we, the collective conscious of Ubuntu, know the answers to, yet it is still difficult to find up-to-date answers to these questions, often requiring input from some of our busiest and most knowledgeable contributors to settle discussions and answer basic queries.

If a potential contributor identifies a bug in a package, there should be one authoritative source of information on where the package source can be located, how it can be pulled, built and tested, and how to work with a sponsor to land changes. Such a process is satisfying for contributors, making it more likely they’ll stay engaged, and therefore benefits the distribution’s longevity and sustainability.

Answering questions and mentoring people will remain a central part of our community’s role, but many of the first questions asked could be serviced by better documentation.

The Challenge #

Much of the required content already exists. The venerable Ubuntu Wiki was the go-to destination for such documentation, but has become outdated both technologically and in the content it serves. This degradation gained pace as we diversified the number of destinations that documentation could live: the Wiki, Discourse, Github, Launchpad, etc.

The Ubuntu Community team have made significant efforts over the past months to centralise the documentation for membership, our code of conduct and project governance. I also called out the renewed Stable Release Update (SRU) documentation in my first post for having made its first steps toward a new and improved structure.

These examples prove we have all the skills we need to write excellent documentation. Throughout the 25.10 cycle, I intend to put some focus on this, consolidating as much of our content into modern formats as possible, thereby making it as accessible as possible.

In doing this work, I hope to:

Illustrate contributor journeys across disciplines
Create resilience in the project by reducing the “bus factor”
Increase the accessibility and ergonomics of our documentation
Enable more efficient, asynchronous collaboration on a wide range of tasks

End Goal #

To quote the Canonical.com page on Documentation Practice:

At the heart of this mission Diátaxis: a way of thinking about documentation. Diátaxis “prescribes approaches to content, architecture and form that emerge from a systematic approach to understanding the needs of documentation users”.

You’ll have seen Diátaxis in use across many of our product documentation pages: the Juju docs, the MAAS docs, the Pebble docs, the Rockcraft docs and many more.

Most of those existing sites are specific - they document a particular product or ecosystem which neatly scopes the documentation structure, but the Diátaxis framework can also be used to bring structure, precision and clarity to the documentation of the Ubuntu project as a whole.

Earlier this month I surveyed the various documentation sites in use by Canonical and the Ubuntu Community, and settled on three common themes around which we will structure our renewed project documentation:

Governance: in which membership, code of conduct, team structures, communication practices, delegation, mission, software licensing and 3rd-party software guidelines will be documented.
Develop Ubuntu: documentation for current and aspiring Ubuntu developers, including how to package software for Ubuntu, how to merge packages from Debian, how to sponsor packages, how to use git-ubuntu and conduct “+1 Maintenance”, etc.
Archive Administration: the nuts and bolts of managing Ubuntu’s prolific software repositories: how to manage seeds, configure phased updates, conduct an MIR, run an SRU process, etc.

These categories were not immediately obvious, and they’re not necessarily mutually exclusive, but they fell out quite naturally when trying to logically organise our existing content.

During the process, I came up with this rough sketch:

This illustrates how multiple categories of documentation from different corners of Ubuntu might come together in a single landing page. To give an idea of how we might further break down existing content by type, then category:

This may not be the final structure, but it’s indicative of how we can use Diátaxis to break down large documentation premises into smaller, more digestible and more ergonomic pieces.

The Plan #

During the Ubuntu 25.10 cycle, we’ll be dedicating two of our Technical Authors to make this happen. One of these authors has been largely responsible for overhauling the Ubuntu Server docs, but both are very familiar with Diátaxis and the tooling we using to deliver documentation.

Throughout this process, we’ll likely come across outdated, poorly reviewed or incorrect documentation, but as we work through the process of consolidating, we can note where this has happened and get it on our backlog to fix.

Perhaps we’ll find items which lend themselves to inclusion in the Canonical Open Documentation Academy, or maybe we’ll need to reach out to some of our less active community members for clarification, but once the structure is in place we’ll at least have a place to collaborate.

Once the transition is complete, there will be an authoritative source for project documentation that is easy to navigate, easy to contribute to and with a well-defined review process that encourages progress over gatekeeping.

Summary #

Documentation is the backbone of a thriving open-source community, guiding contributors, setting expectations, and ensuring long-term sustainability.

While Ubuntu has extensive documentation, much of it is scattered, outdated, or difficult to navigate. By leveraging the Diátaxis framework, we aim to bring structure, clarity, and accessibility to Ubuntu Project Documentation. Our focus will be on governance, development, and archive administration, ensuring that key processes and responsibilities are well-documented and easy to follow.

With dedicated technical authors and community collaboration, the Ubuntu 25.10 cycle will mark a significant step toward making our documentation searchable, structured, and sustainable.

I hope this effort will empower contributors, reduce reliance on institutional knowledge, and create a more resilient project for the next generation of Ubuntu developers and users.

Carefully But Purposefully Oxidising Ubuntu

Wed, 12 Mar 2025 00:00:00 +0000

Introduction #

Last month I published Engineering Ubuntu For The Next 20 Years, which outlines four key themes for how I intend to evolve Ubuntu in the coming years. In this post, I’ll focus on “Modernisation”. There are many areas we could look to modernise in Ubuntu: we could focus on the graphical shell experience, the virtualisation stack, core system utilities, default shell utilities, etc.

Over the years, projects like GNU Coreutils have been instrumental in shaping the Unix-like experience that Ubuntu and other Linux distributions ship to millions of users. According to the GNU website:

This package provides utilities which have become synonymous with Linux to many - the likes of ls, cp, and mv. In recent years, there has been an effort to reimplement this suite of tools in Rust, with the goal of reaching 100% compatibility with the existing tools. Similar projects, like sudo-rs, aim to replace key security-critical utilities with more modern, memory-safe alternatives.

Starting with Ubuntu 25.10, my goal is to adopt some of these modern implementations as the default. My immediate goal is to make uutils’ coreutils implementation the default in Ubuntu 25.10, and subsequently in our next Long Term Support (LTS) release, Ubuntu 26.04 LTS, if the conditions are right.

But… why? #

Performance is a frequently cited rationale for “Rewrite it in Rust” projects. While performance is high on my list of priorities, it’s not the primary driver behind this change. These utilities are at the heart of the distribution - and it’s the enhanced resilience and safety that is more easily achieved with Rust ports that are most attractive to me.

The Rust language, its type system and its borrow checker (and its community!) work together to encourage developers to write safe, sound, resilient software. With added safety comes an increase in security guarantees, and with an increase in security comes an increase in overall resilience of the system - and where better to start than with the foundational tools that build the distribution?

I recently read an article about targeting foundational software with Rust in 2025. Among other things, the article asserts that “foundational software needs performance, reliability — and productivity”. If foundational software fails, so do all of the other layers built on top. If foundational packages have performance bottlenecks, they become a floor on the performance achievable by the layers above.

Ubuntu powers millions of devices around the world, from servers in your data centre, to safety critical systems in autonomous systems, so it behooves us to be absolutely certain we’re shipping the most resilient and trustworthy software we can.

There are lots of ways to achieve this: we can provide long term support for projects like Kubernetes, we can assure the code we write, and we can strive to achieve compliance with safety-centric standards, but another is by shipping software with the values of safety, soundness, correctness and resilience at their core.

That’s not to throw shade on the existing implementations, of course. Many of these tools have been stable for many years, quietly improving performance and fixing bugs. A lovely side benefit of working on newer implementations, is that it sometimes facilitates improvements in the original upstream projects, too!

I’ve written about my desire to increase the number of Ubuntu contributors, and I think projects like this will help. Rust may present a steeper learning curve than C in some ways, but by providing such a strong framework around the use of memory it also lowers the chances that a contributor accidentally commits potentially unsafe code.

Introducing `oxidizr` #

I did my homework before writing this post. I wanted to see how easy it was for me to live with these newer implementations and get a sense of their readiness for prime-time within the distribution. I also wanted a means of toggling between implementations so that I could easily switch back should I run into incompatibilities - and so oxidizr was born!

The oxidizr utility enables you to quickly swap in and out newer implementations of certain packages with relatively low risk. It has the notion of Experiments, where each experiment is a package that already exists in the archive that can be swapped in as an alternative to the default.

Version 1.0.0 supports the following experiments:

How does it work? #

Each experiment is subtly different since the paths of the utilities being replaced vary, but the process for enabling an experiment is generally:

Install the alternative package (e.g. apt install rust-coreutils)
For each binary shipped in the new package:
- Lookup the default path for that utility (e.g which date)
- Back up that file (e.g. cp /usr/bin/date /usr/bin/.date.oxidizr.bak)
- Symlink the new implementation in place (e.g. ln -s /usr/bin/coreutils /usr/bin/date)

There is also the facility to “disable” an experiment, which does the reverse of the sequence above:

For each binary shipped in the new package:
- Lookup the default path for the utility (e.g which date)
- Check for and restore any backed up versions (e.g cp /usr/bin/.date.oxidizr.bak /usr/bin/date)
Uninstall the package (e.g. apt remove rust-coreutils)

Thereby returning the system back to its original state! The tool is covered by a suite of integration tests which illustrate this behaviour which you can find on Github

Get started #

There are a couple of ways to get oxidizr on your system. If you already use cargo, you can do the following:

1

cargo install --git https://github.com/jnsgruk/oxidizr

Otherwise, you can download and install binary releases from Github:

1
2


# Download version 1.0.0 and extract to /usr/bin/oxidizr
curl -sL "https://github.com/jnsgruk/oxidizr/releases/download/v1.0.0/oxidizr_Linux_$(uname -m).tar.gz" | sudo tar -xvzf - -C /usr/bin oxidizr

Once installed you can invoke oxidizr to selectively enable/disable experiments. The default set of experiments in v1.0.0 is rust-coreutils and sudo-rs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Enable default experiments
sudo oxidizr enable
# Disable default experiments
sudo oxidizr disable
# Enable just coreutils
sudo oxidizr enable --experiments coreutils
# Enable all experiments without prompting with debug logging enabled
sudo oxidizr enable --all --yes -v
# Disable all experiments without prompting
sudo oxidizr disable --all --yes

The tool should work on all versions of Ubuntu after 24.04 LTS - though the diffutils experiment is only available from Ubuntu 24.10 onward.

The tool itself is stable and well covered with unit and integration tests, but nonetheless I’d urge you to start with a test virtual machine or a machine that isn’t your production workstation or server! I’ve been running the coreutils and sudo-rs experiments for around 2 weeks now on my Ubuntu 24.10 machines and haven’t had many issues (more on that below…).

How to Help #

If you’re interested in helping out on this mission, then I’d encourage you to play with the packages, either by installing them yourself or using oxidizr. Reply to the Discourse post with your experiences, file bugs and perhaps even dedicate some time to the relevant upstream projects to help with resolving bugs, implementing features or improving documentation, depending on your skill set.

You can also join us to discuss on our Matrix instance.

Next Steps #

Earlier this week, I met with @sylvestre to discuss my proposal to make uutils coreutils the default in Ubuntu 25.10. I was pleased to hear that he feels the project is ready for that level of exposure, so now we just need to work out the specifics. The Ubuntu Foundations team is already working up a plan for next cycle.

There will certainly be a few rough edges we’ll need to work out. In my testing, for example, the only incompatibility I’ve come across is that the update-initramfs script for Ubuntu uses cp -Z to preserve selinux labels when copying files. The cp, mv and ls commands from uutils don’t yet support the -Z flag, but I think we’ve worked out a way to unblock that work going forward, both in the upstream and in the next release of Ubuntu.

I’m going to do some more digging on sudo-rs over the coming weeks, with a view to assessing a similar transition.

Summary #

I’m really excited to see so much investment in the foundational utilities behind Linux. The uutils project seems to be picking up speed after their recent appearance at FOSDEM 2025, with efforts ongoing to rework procps, util-linux and more.

The sudo-rs project is now maintained by the Trifecta Tech Foundation, who are focused on “open infrastructure software in the public interest” . Their zlib-rs recently released v0.4.2, which appears to now be the fastest API-compatible zlib implementation. They’re also behind the Pendulum Project and ntpd-rs for memory-safe time synchronisation.

With Ubuntu, we’re in a position to drive awareness and adoption of these modern equivalents by making them either trivially available, or the default implementation for the world’s most deployed Linux distribution.

We will need to do so carefully, and be willing to scale back on the ambition where appropriate to avoid diluting the promise of stability and reliability that the Ubuntu LTS releases have become known for, but I’m confident that we can make progress on these topics over the coming months.

Engineering Ubuntu For The Next 20 Years

Tue, 11 Feb 2025 00:00:00 +0000

Introduction #

I’ve been a VP Engineering at Canonical for 3 years now, building Juju and our catalog of charms. In the last week of January, I was appointed the VP Engineering for Ubuntu at Canonical, where I will now oversee the Ubuntu Foundations, Server and Desktop teams.

Over the past 20 years, Ubuntu has become synonymous with “Linux” to many people. I fondly remember receiving my first Ubuntu CD in the post, shortly after my own Linux journey began in 2003 with booting Knoppix on a school computer. Throughout my career, Linux and open source have been prominent features that I’m very proud of. In the past few years I’ve made contributions to Ubuntu, Arch Linux, and more recently NixOS.

Ubuntu’s recent 20 year milestone is a timely reminder to pause and reflect on what made Ubuntu so exciting, so successful and so captivating to the Linux community. In 2004, the idea of releasing an operating system every six months was laughed off by many, but has now become the norm. Ubuntu builds upon Debian, aiming to bring the latest and very best open source had to offer to the masses. In the past 10 years, we’ve seen huge shifts in the way software is delivered - the success of large-scale cloud based operations necessitated a shift towards more automated testing, releasing and monitoring, and as the open source community around these projects grew, we had to evolve our ways of thinking, designing and communicating about software.

Four Key Themes #

As I step into this new role, I’ve reflected on how we can steer the engineering efforts behind Ubuntu. I’ve anchored this vision around four themes: Communication, Automation, Process and Modernisation.

Communication #

Communication is a central component of a distributed workforce - whether that workforce is employed by Canonical, members of our community or contributors from our partners. Ubuntu has relied for many years on mailing lists and IRC. These platforms enabled global teams to collaborate for years, and have been invaluable to the community. In 2025 we’re fortunate to have a wealth of communications platforms at our disposal, but we must use these tools strategically to avoid fragmentation.

On Jan 29 2025, the Ubuntu developer mailing list announced that the primary means of communication for Ubuntu developers will be the Ubuntu Community Matrix server. Matrix provides a rich, modern communications medium that is familiar to the next generation of engineers and tinkerers, who will be central to the continued progression of Ubuntu and open source. We’re in good company on Matrix, with many other Linux distributions and projects maintaining a presence on the platform. The recent migration of Ubuntu Forums to the Ubuntu Discourse, further consolidates the range of platforms we use to connect with one another.

To effect much of the change I’m describing in this post, we will need community support. I’ll be encouraging the leads of our internal teams in Ubuntu Foundations, Server and Desktop to be more forthcoming and regular with public updates that will serve two purposes: to share our intentions, progress and dreams for Ubuntu, but also to collaborate on refining our vision, ensuring we deliver a platform that is not just exciting, but relevant for many years to come.

Documentation is a critical form of communication. Our documentation enables our current users, but also illuminates the path for new contributors. Such documentation does exist, but much of it is fragmented across different platforms, duplicated and/or contradictory or simply difficult to find. As a company, and as a community, we must focus on ensuring both existing and potential contributors have access to the information they need on conventions, tools and processes. A good example of where this has already happened is the SRU documentation which was recently rebuilt in line with our documentation practices.

Automation #

Delivering a Linux distribution is a monumental task. With tens of thousands of packages across multiple architectures, the workload can be overwhelming - leaving little room for innovation until the foundational work is done. We’re fortunate to benefit from the diligent work done by the Debian community, yet there is a huge amount of work that goes into each Ubuntu release. One of our primary tasks as a distribution is package maintenance. While some may see this as menial or repetitive, it remains critical to the future of Ubuntu, and is a valuable specialist skill in its own right.

Software packaging is a complex and constantly evolving topic. Ubuntu relies heavily on a blend of Debian packages, and our own Snap packaging format. Debian packaging was revolutionary - responsible for huge advancements in the way we thought about delivering software, but as things have moved on some of those tools and practices are beginning to show their age.

I’d like to focus on enriching our build process with modern ideals and processes for automating the version bumps, testing, performance benchmarking and releasing of packages in the archive. High complexity tasks are error-prone and, without sufficient automation, risk becoming overly dependent on a few skilled individuals. We have the same challenge with Snaps, but they benefit from significantly more modern tooling as a consequence of the observations made about Debian packaging over many years.

The goal of this theme is not just to automate as much as possible (thereby increasing our collective capacity), but also to simplify processes where we can. Much of Ubuntu’s build process *is* automated, but those systems are disparate and often opaque to all but our most experienced contributors.

I’ve been inspired by how the NixOS community manages packaging. Every single package for the distro is represented as text files, in a single Git repository, with a universally observable continuous integration and integration testing pipeline (Hydra) that performs version bumps and simple maintenance tasks semi-autonomously. While this model carries its own challenges, there is something alluring about the transparency and accessibility of the systems that assemble, test and deliver software to their users.

Universal Blue, and by extension Project Bluefin, are recent additions to the Linux ecosystem that benefited from thinking hard about the tooling they use to build their distribution. They’ve centered their process around tools with which their cloud-native audience are already familiar.

My suggestion is not to imitate these projects, rather that the open source community is at its strongest when we collaborate and learn from one another. I think we can take inspiration from those surrounding us, and use that to inform our plans for Ubuntu’s future.

Process #

Process is closely tied to automation, but is frequently viewed negatively in software engineering, carrying connotations of bureaucracy and slowdowns. In my experience, a well-designed process empowers people to enact changes with confidence.

Ubuntu is built by all of us, in many countries and across all timezones. Concise, well-defined, lightweight processes promote autonomy and reduce uncertainty - enabling people to unblock themselves. Ubuntu is no stranger to process: the Main Inclusion Review (MIR), the aforementioned Stable Release Updates (SRU) process, the process for Snap store requests and many more have contributed to the success of Ubuntu, setting clear guardrails for contributors and ensuring we work to common standards.

My goal over the coming months is to work with you, the people behind Ubuntu, to identify which of these processes still serve us, and which need revising to simplify our work while maintaining our dedication to stability. I’ll consolidate the definitions of these processes, make them searchable, peer-reviewable, and more discoverable. Examples of where this has worked well are the Go proposal process, and the Ethereum Improvement Proposal process - both of which make it trivial to create, track and discuss proposals across the breadth of their respective projects.

If you submit an MIR, or work on an SRU, it should be trivial to understand the status of that request, and to communicate with the team executing that process where needed. If you’re interested in joining our community, it should be simple to get a sense of what is changing across the project, and where you might be able to help.

I’d like to tackle these problems and make these processes as transparent as possible.

Modernisation #

The world of computing has evolved dramatically in the last 20 years, and I’m proud that Ubuntu has continually adapted and thrived. In Linux alone there have been huge changes to what is considered “normal” for a Linux machine. Whether it be the introduction of `systemd`, the advent of languages with a focus on memory safety, the huge growth in virtualisation and containerisation technology, or even the introduction of Rust into the Linux kernel itself - the foundations of our distribution must be constantly assessed against the needs of our users.

I was proud to see the announcement last year that the Ubuntu Kernel team committed to shipping the very latest kernels in new versions of Ubuntu, wherever they possibly can. Even if that means shipping a kernel that’s in the release candidate phase, the team will stand by that kernel and continue to support it through the Ubuntu release’s life. While this could appear cavalier at a glance, what it represents is a willingness to rise to the challenge of shipping the very best of open source to our users. I’d like to see more of this. Ubuntu is a flagship Linux distribution and a starting point for many; we must ensure that our users are presented with the very best our community has to offer - even if that means a bit more hustle in the early days of a given release. This is of particular importance for our Long Term Support releases, which are relied upon by governments, financial institutions, educational establishments, nonprofits and many others for years after the initial release date.

We should look deeply at the tools we ship with Ubuntu by default - selecting for tools that have resilience, performance and maintainability at their core. There are countless examples in the open source community of tools being re-engineered, and re-imagined using tools and practices that have only relatively recently become available. Some of my personal favourites include command-line utilities such as eza, bat, and helix, the new ghostty terminal emulator, and more foundational projects such as the uutils rewrite of coreutils in Rust. Each of these projects are at varying levels of maturity, but have demonstrated a vision for a more modern Unix-like experience that emphasises resilience, performance and usability.

Another example of this is our work on TPM-backed full disk encryption, a project which promises encryption of our users’ data with no degradation to their user experience. This feature relies upon cryptographic hardware and techniques that have only recently become available to us, but enable us to deliver the potent combination of security and usability to our users.

Delivering Features #

What I’ve shared so far is a high-level overview, and many of the points under the four themes will take time to implement, with most appearing as a series of gradual improvements. You might be wondering whether we’ll focus on the latest trends and features, or prioritise that bug you reported.

While focusing on the latest trends or a single breakthrough feature can yield short-term progress, embracing these principles will create the space for sustained, impactful innovation.

That said, I’ve also been working on a list of incremental features and improvements that we can deliver in the coming months to enhance the Ubuntu experience. You’ll hear more from me and the team leads regularly as we share updates and progress.

Summary #

I’m incredibly excited to embark on this journey, and consider it a privilege to serve in this role. Together with the Ubuntu community, Canonical engineers, and our partners, we will build an open-source platform that enables the next 20 years of innovation in computing.

If you have ideas for the future of Ubuntu, or something in this post has resonated with you and you want to be involved either as a community member, or perhaps a future employee of Canonical, I’d love to hear from you.

Workstation VMs with LXD & Multipass

Tue, 25 Jun 2024 00:00:00 +0000

Introduction #

Over the years, I’ve used countless tools for creating virtual machines - often just for short periods of time when testing new software, trying out a new desktop environment, or creating a more isolated development environment. I’ve gone from just using the venerable qemu at the command line, to full-blown desktop applications like Virtualbox, to using virt-manager with libvirt.

When I joined Canonical back in March 2021, I’d hardly used LXD, and I hadn’t ever used Multipass. Since then, they’ve both become indispensable parts of my workflow, so I thought I’d share why I like them, and how I use each of them in my day to day work.

I work for Canonical, and am therefore invested in the success of their products, but at the time of writing I’m not responsible for either LXD or Multipass, and this post represents my honest opinions as a user of the products, and nothing more.

Installation / Distribution #

Both LXD and Multipass are available as snap packages, and that’s the most supported and recommended route for installation. LXD is available in the repos of a few other Linux distributions (including NixOS, Arch Linux), but the snap package also works great on Arch, Fedora, etc. I personally ran Multipass and LXD as snaps on Arch Linux for a couple of years without issue.

If you’d like to follow along with the commands in this post, you can get setup like so:

1
2
3
4
5
6
7
8
9


sudo snap install lxd
sudo lxd init --minimal

# If you'd like to use LXD/LXC commands without sudo
# run the following command and logout/login:
#
# sudo usermod -aG lxd $USER

sudo snap install multipass

Early on in my journey with NixOS, I packaged Multipass for Nix. I still maintain (and use!) the NixOS module. This was my first ever contribution to NixOS – a fairly colourful review process to say the least…

The result is that you can use something like the following in your configuration, and have multipass be available to you after a nixos-rebuild switch:

1
2
3


{
 virtualisation.multipass.enable = true;
}

LXD has been maintained in NixOS for many years now - and around this time last year I added support for the LXD UI. The screenshots you see throughout this post are all from LXD UI running on a NixOS machine using the following configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


{
 virtualisation.lxd = {
 enable = true;
 zfsSupport = true;
 ui.enable = true;
 };

 networking = {
 firewall = {
 trustedInterfaces = [ "lxdbr0" ];
 };
 };
}

Ubuntu on-demand with Multipass #

Multipass is designed to provide simple on-demand access to Ubuntu VMs from any workstation - whether that workstation is running Linux, macOS or Windows. It is designed to replicate, in a lightweight way, the experience of provisioning a simple Ubuntu VM on a cloud.

Multipass makes use of whichever the most appropriate hypervisor is on a given platform. On Linux, it can use QEMU, LXD or libvirt as backends, on Windows it can use Hyper-V or Virtualbox, and on macOS it can use QEMU or Virtualbox. Multipass refers to these backends as drivers.

Multipass’ scope is relatively limited, but in my opinion that’s what makes it so delightful to use. Once installed, the basic operation of Multipass couldn’t be simpler:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


❯ multipass shell
Launched: primary
Mounted '/home/jon' into 'primary:Home'
Welcome to Ubuntu 24.04 LTS (GNU/Linux 6.8.0-35-generic x86_64)

 * Documentation: https://help.ubuntu.com
 * Management: https://landscape.canonical.com
 * Support: https://ubuntu.com/pro

 System information as of Tue Jun 25 11:17:55 BST 2024

 System load: 0.4 Processes: 132
 Usage of /: 38.9% of 3.80GB Users logged in: 0
 Memory usage: 31% IPv4 address for ens3: 10.93.253.20
 Swap usage: 0%


Expanded Security Maintenance for Applications is not enabled.

3 updates can be applied immediately.
1 of these updates is a standard security update.
To see these additional updates run: apt list --upgradable

Enable ESM Apps to receive additional future security updates.
See https://ubuntu.com/esm or run: sudo pro status


ubuntu@primary:~$

This one command will take care of creating the primary instance if it doesn’t already exist, start the instance and drop you into a bash shell - normally in under a minute.

Multipass has a neat trick: it bundles a reverse SSHFS server that enables easy mounting of the host’s home directory into the VM. This happens by default for the primary instance. As a result the instance I created above has my home directory mounted at /home/ubuntu/Home - making it trivial to jump between editing code/files on my host and in the VM. I find this really useful - I can edit files on my workstation in my own editor, using my Yubikey to sign and push commits without having to worry about complicated provisioning or passthrough to the VM, and any files resulting from a build process on my workstation are instantly available in the VM for testing.

Multipass instances can be customised a little. You won’t find complicated features like PCI-passthrough, but basic parameters can be tweaked. The commands I usually run for setting up a development machine when I’m working on Juju/Charms are:

1
2
3
4
5
6


# Create a machine named 'dev' with 16 cores, 40GiB RAM and 100GiB disk
multipass launch noble -n dev -c 16 -m 40G -d 100G
# Mount my home directory into the VM
multipass mount /home/jon dev:/home/ubuntu/Home
# Get a shell in the VM
multipass shell dev

Once you’re done with an instance, you can remove it like so:

1
2


multipass remove dev
multipass purge

Multipass does have some more interesting features, though most of my usage is represented above. One feature that might be of more interest for MacOS or Windows users is aliases. This feature enables you to alias local commands to their counterparts in a Multipass VM, meaning for example that every time you run docker on your Mac, the command is actually executed inside the Multipass VM:

1
2
3


# Example of mapping the local `mdocker` command -> `docker` in
# the multipass VM
multipass alias dev:docker mdocker

Multipass will launch the latest Ubuntu LTS by default, but there are a number of other images available - including some “appliance” images for applications like Nextcloud, Mosquitto, etc.

There is also the concept of Blueprints which are essentially recipes for virtual machines with a given purpose. These are curated partly by the Multipass team, and partly by the community. A blueprint enables the author to specify cores, memory, disk, cloud-init data, aliases, health checks and more. The recipes themselves are maintained on Github, and you can see the list of available images/blueprints using multipass find:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


❯ multipass find
Image Aliases Version Description
core core16 20200818 Ubuntu Core 16
core18 20211124 Ubuntu Core 18
core20 20230119 Ubuntu Core 20
core22 20230717 Ubuntu Core 22
20.04 focal 20240612 Ubuntu 20.04 LTS
22.04 jammy 20240614 Ubuntu 22.04 LTS
23.10 mantic 20240619 Ubuntu 23.10
24.04 noble,lts 20240622 Ubuntu 24.04 LTS
daily:24.10 oracular,devel 20240622 Ubuntu 24.10
appliance:adguard-home 20200812 Ubuntu AdGuard Home Appliance
appliance:mosquitto 20200812 Ubuntu Mosquitto Appliance
appliance:nextcloud 20200812 Ubuntu Nextcloud Appliance
appliance:openhab 20200812 Ubuntu openHAB Home Appliance
appliance:plexmediaserver 20200812 Ubuntu Plex Media Server Appliance

Blueprint Aliases Version Description
anbox-cloud-appliance latest Anbox Cloud Appliance
charm-dev latest A development and testing environment for charmers
docker 0.4 A Docker environment with Portainer and related tools
jellyfin latest Jellyfin is a Free Software Media System that puts you in control of managing and streaming your media.
minikube latest minikube is local Kubernetes
ros-noetic 0.1 A development and testing environment for ROS Noetic.
ros2-humble 0.1 A development and testing environment for ROS 2 Humble.

The team also recently introduced the ability to snapshot virtual machines, though I must confess I’ve not tried it out in anger yet.

LXD… for VMs? #

For many people, LXD is a container manager - and indeed for many years it could “only” manage containers. LXD was built for running “system containers”, as opposed to “application containers” like Docker/Podman (or Kubernetes). Running a container with LXD is more similar to to running a container with systemd-nspawn, but with the added bonus that it can cluster across machines, authenticate against different identity backends, and manage more sophisticated storage.

Because LXD manages system containers, each container gets its own systemd, and behaves more like a ’lightweight VM’ sharing the host’s kernel. This turns out to be a very interesting property for people who want to get some of the benefits of containerisation (i.e. higher workload density, easier snapshotting, migration, etc.) with more legacy applications that might struggle to run effectively in application containers.

But this post is about virtual machines. Since the 4.0 LTS release, LXD has also supported running VMs with qemu. The API for launching a container is identical to launching a virtual machine. Better still, Canonical provides images for lots of different Linux distributions, and even desktop variants of some images - meaning you can quickly get up and running with a wide range of distributions, for example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# Launch a Ubuntu 24.04 LTS VM
lxc launch ubuntu:noble ubuntu --vm
# Get a shell inside the VM
lxc exec ubuntu bash

# Launch a Fedora 40 VM
lxc launch images:fedora/40 fedora --vm
# Get a shell inside the VM
lxc exec fedora bash

# Launch an Arch Linux VM (doesn't support secure boot yet)
lxc launch images:archlinux arch --vm -c security.secureboot=false
# Get a shell inside the VM
lxc exec arch bash

You can get a full list of virtual machine images like so:

1

lxc image ls images: --format=compact | grep VIRTUAL-MACHINE

LXD Desktop VMs #

Another neat trick for LXD is desktop virtual machines. These are launched with curated images that drop you into a minimal desktop environment that’s configured to automatically login. This has to be one of my favourite features of LXD!

1
2


# Launch a Ubuntu 24.04 LTS desktop VM and get a console
lxc launch images:ubuntu/24.04/desktop ubuntu --vm --console=vga

The guest is pre-configured to work correctly with SPICE, so that means clipboard integration, automatic resizing with the viewer window, USB redirection, etc. The same also works for other distros, as before:

1
2
3
4
5
6
7
8


# Launch an Arch desktop VM
lxc launch images:archlinux/desktop-gnome arch --vm \
 -c limits.cpu=8 \
 -c limits.memory=16GiB \
 -c security.secureboot=false

# Get a console using a separate command (if preferred!)
lxc console --type=vga arch

LXD UI 😍 #

Back in June 2023, Canonical announced early access to the LXD graphical user interface on their blog. The LXD UI is now generally available and enabled by default from LXD 5.21 onwards - though you can find instructions for enabling it on earlier versions in the docs. The summary is:

1
2
3
4


lxc config set core.https_address :8443

sudo snap set lxd ui.enable=true
sudo systemctl reload snap.lxd.daemon

In my opinion, the LXD UI is one of the best, if not the best way to interact with a hypervisor yet. Being a full-stack web application, it gains independence from different GUI toolkits on Linux and, provided the cluster is remote, can be accessed the same way from Windows, Mac and Linux.

I’ve used other hypervisors with web UIs, particularly Proxmox, and I’ve found the experience with LXD UI to be very smooth, even from the early days. The UI can walk you through the creation and management of VMs, containers, storage and networking. The UI can also give you a nice concise summary of each instance (below is the summary of the VM created using the command in the last section):

One of my favourite features is the web-based SPICE console for desktop VMs, which combined with the management features makes it trivial to stand up a desktop VM and start testing:

Why both? #

By now you’ve probably realised that LXD can do everything Multipass can do, and give much more flexibility - and that’s true. LXD is a full-featured hypervisor which supports much more sophisticated networking, PCI-passthrough, clustering, integration with enterprise identity providers, observability through Prometheus metrics and Loki log-forwarding, etc.

Multipass is small, lean and very easy to configure. If I just want a quick command-line only Ubuntu VM to play with, I still find multipass shell to be most convenient - especially with the automatic home directory mounting.

When I want to work with desktop VMs, interact with non-Ubuntu distributions, or work more closely with hardware, then I use LXD. I was already a bit of a closet LXD fan, having previously described it as a bit of a “secret weapon” for Canonical, but since the introduction of the LXD UI, I’m a fully paid up member of the LXD fan club 😉

Summary #

As I mentioned in the opening paragraphs - both LXD and Multipass have become central to a lot of my technical workflows. The reason I packaged Multipass for NixOS, was that I wanted to dive into daily-driving NixOS, but not without Multipass! In my opinion, the LXD UI is one of the most polished experiences for managing containers and VMs on Linux, and I’m really excited for what that team cooks up next.

Canonical on Jon Seager

ntpd-rs: it's about time!

Introduction #

NTP, NTS, and PTP #

Proven at Scale #

A Single, Memory-Safe Utility for NTP and PTP #

Timelines and Goals #

About the Trifecta Tech Foundation #

Summary #

An update on upki

Architecture & Progress #

How to try it #

Next Steps #

Further Down the Road #

Certificate Transparency Enforcement #

Intermediate Preloading #

Merkle Tree Certificates #

Summary #

Developing with AI on Ubuntu

Hardware & Drivers #

Inference Snaps #

Sandboxing Agents #

My agent sandboxes itself! #

Sandbox with LXD containers #

Sandbox with LXD VMs #

Sandbox with Multipass #

Sandbox with WSL #

Summary #

Addressing Linux's Missing PKI Infrastructure

The Problem #

CRLite #

Introducing upki #

Ecosystem Compatibility #

Timeline #

Summary #

Ubuntu Summit 25.10: Personal Highlights

DOOM in Space #

Infrastructure-Wide Profiling of Nvidia CUDA #

Inference Snaps #

Nøughty Linux: Ubuntu’s Stability Meets Nixpkgs’ Freshness #

Are we stuck with the same Desktop UX forever? #

Honorable Mentions #

Conclusion #

Ubuntu Engineering in 2025: A Retrospective

Ubuntu 25.10: A Retrospective #

Communication #

Automation #

Process #

Modernisation #

What’s Next? #

The Immutable Linux Paradox

What is an immutable Linux distribution? #

Why immutability? #

The immutability paradox #

Approaches to immutability #

Fedora Silverblue / CoreOS / EndlessOS (ostree) #

RHEL “Image Mode” (bootc) #

NixOS #

Ubuntu Core #

Summary #

Crafting Your Software

Software build lifecycle #

Lifecycle stages #

Lifecycle in the CLI #

Part definition #

Isolated build environments #

Saving space #

Multi-architecture builds #

Unified testing framework #

Crafting the crafts #

Examples and docs #

Summary #

Introducing Debcrafters

Bootstrapping the team #

Mission #

Tools & processes #

Attracting contributors #

Contribution beyond Ubuntu #

Summary #

Supercharging Ubuntu Releases: Monthly Snapshots & Automation

Fedora Silverblue / CoreOS / EndlessOS (`ostree`) #

RHEL “Image Mode” (`bootc`) #

What is `sudo-rs`? #

Progress on `coreutils` #

Introducing `oxidizr` #