Security on Jon Seager

ntpd-rs: it's about time!

Thu, 26 Mar 2026 00:00:00 +0000

This article was originally posted on the Ubuntu Discourse, and is reposted here. I welcome comments and further discussion in that thread.

Introduction #

I am thrilled to announce the next target in our campaign to replace core system utilities with memory-safe Rust rewrites in Ubuntu. In upcoming releases, Ubuntu will be adopting ntpd-rs as the default time synchronization client and server, eventually replacing chrony, linuxptp and with any luck, gpsd for time syncing use-cases.

ntpd-rs is a full-featured implementation of the Network Time Protocol (NTP), written entirely in Rust. Maintained by the Trifecta Tech Foundation as part of Project Pendulum, ntpd-rs places a strong focus on security, stability, and memory safety.

To deliver on this goal, we’re building on our partnership with the Trifecta Tech Foundation who are behind sudo-rs, zlib-rs and more. We will be funding the Trifecta Tech Foundation to build new features, enhance security isolation, and ultimately deliver a unified, memory-safe time synchronization utility for the Linux ecosystem. This work meshes well with the Trifecta Tech Foundations goals to improve the security of time synchronization everywhere.

NTP, NTS, and PTP #

Before diving into the mechanics and reasoning behind the transition, I wanted to give some background on the protocols at play, and the problems we’re hoping to solve. Keeping accurate time is a critical system function, not least because it involves constant interaction with the internet and forms the basis for cryptographic verification in protocols such as Transport Layer Security (TLS).

NTP (Network Time Protocol) is the foundational protocol that most operating systems implement to accurately determine the current time from a network source.

NTS (Network Time Security) is to NTP what HTTPS is to HTTP. Historically, the Network Time Protocol was used unencrypted, like many of the early web protocols. NTS introduces cryptographic security to time synchronization, ensuring that bad actors cannot intercept or spoof time data. We already pushed to make NTS the default out-of-the-box in Ubuntu 25.10, which we accomplished by migrating away from ntpd to chrony as the default time-syncing implementation.

PTP (Precision Time Protocol) is used for systems that require sub-microsecond synchronization. While the precision offered by a standard NTP deployment is sufficient for general-purpose computing, PTP is often used for complex, specialized deployments like telecommunications networks, power grids, and automotive applications.

Proven at Scale #

Transitioning core utilities in Ubuntu comes with a responsibility to ensure that replacements are of high quality, resilient and offer something to the platform. We may be the first major Linux distribution to adopt ntpd-rs by default, but we aren’t the first to recognize the readiness of ntpd-rs - it has already been proven at scale by Let’s Encrypt.

While Let’s Encrypt’s core Certificate Authority software has always been written in memory-safe Go, their server operating systems and network infrastructure historically relied on memory-unsafe languages like C and C++, which routinely led to vulnerabilities requiring patching.

Following extensive development, ntpd-rs was deployed to Let’s Encrypt’s staging environment in April 2024, and rolled out to full production by June 2024, marking a major milestone for ntpd-rs.

The fact that one of the world’s most prolific and security-conscious certificate authorities trusts ntpd-rs to keep time across its fleet should provide us, and our enterprise customers, with tremendous confidence in its resilience and suitability.

A Single, Memory-Safe Utility for NTP and PTP #

We want to provide a single utility for configuring both NTP/NTS and Precision Time Protocol (PTP) on Linux. The Trifecta Tech Foundation is concurrently developing Statime, a memory-safe PTP implementation that delivers synchronization performance on par with linuxptp, but with the goal of being easier to configure and use.

The goal is to integrate Statime’s PTP capabilities directly into ntpd-rs, improving the user experience by bringing all time synchronization concerns into one utility with common configuration and usage patterns, obviating the need for complex manual configuration (and troubleshooting) that users of linuxptp may be familiar with.

Timelines and Goals #

As with our transition to sudo-rs and uutils coreutils, leading the mainstream adoption of foundational system utilities comes with responsibility. We want to ensure that ntpd-rs matches the security isolation and performance standards our users expect from chrony.

Canonical is funding the Trifecta Tech Foundation’s development efforts toward these goals over the coming cycles. This work will take place between July 2026 and January 2027 in several major milestones. Our current timeline and targeted goals are:

Ubuntu 26.10: If all goes well, we aim to land the latest version of ntpd-rs in the archive, making it available to test.
Ubuntu 27.04: By 27.04, ntpd-rs should have integrated statime, and we will ship the unified client/server binary for NTP, NTS and PTP in Ubuntu by default, with the aim of providing a smooth migration path for those who already manage complex chrony configs.

To get us there, the Trifecta Tech Foundation will be working on the following items:

Feature Parity & Hardware Support: Adding gpsd IP socket support, multi-threading support for NTP servers, and support for multi-homed servers.
Security & Isolation: chrony is isolated via AppArmor and seccomp. We’ll be working on robust AppArmor and seccomp profiles for ntpd-rs to ensure we don’t buy memory safety at the cost of system-level privilege boundaries. We are also ensuring rustls can use openssl as a crypto provider to satisfy strict corporate cryptography policies.
PTP & Automotive Profiles: Adding support for gPTP, which will allow us to support complex deployments like the Automotive profile directly from nptd-rs (via Statime). Additionally, experimental support for the proposed Client-Server PTP protocol (CSPTP, IEEE 1588.1) will be added.
Benchmarking & Testing: Comprehensive benchmarking of long-term memory, CPU usage, and synchronization performance against chrony to give our cloud partners and enterprise users complete confidence in the transition.
User-experience: Logging improvements and enhancements to configuration that help users configure the time synchronisation target to optimise network usage, as well as improvements to the ntp-cli

About the Trifecta Tech Foundation #

Trifecta Tech Foundation is a non-profit and a Public Benefit Organisation (501(c)(3) equivalent) that creates open-source building blocks for critical infrastructure software. Their initiatives on data compression, time synchronization, and privilege boundary, impact the digital security of millions of people. If you’d like to support their work, please contact them via https://trifectatech.org/support.

Summary #

I am really excited to deepen our already productive relationship with the Trifecta Tech Foundation to make these transitions viable for the wider ecosystem. We’ll be working hard on testing and integration to ensure seamless migration paths, and heavily document the changes ahead of the 26.10 and 27.04 releases.

Stay tuned!

An update on upki

Mon, 16 Feb 2026 00:00:00 +0000

This article was originally posted on the Ubuntu Discourse, and is reposted here. I welcome comments and further discussion in that thread.

Last year, I announced that Canonical had begun supporting the development of upki, a project that will bring browser-grade Public Key Infrastructure (PKI) to Linux. Since then, development has been moving at pace thanks to the tireless work of Dirkjan and Joe.

In this post, I’ll explore the progress we’ve made, how you can try an early version, and where we’re going next.

Architecture & Progress #

As a reminder, upki’s primary goal is to provide a reliable, privacy-preserving, and efficient certificate revocation mechanism for Linux system utilities, package managers, and language runtimes. The solution is built around CRLite, an efficient data format that compresses and distributes certificate revocation information at scale.

The upki repository is structured as a Cargo workspace containing five crates, each serving a distinct role:

upki: the core library and CLI tool. This crate contains the revocation query engine, the client-side sync logic for fetching filter updates, and the command-line interface. The revocation interface was originally embedded in the CLI, but has since been promoted into the library so that other Rust projects can use it directly as a dependency.
upki-mirror: the server-side mirroring tool. This binary fetches and validates CRLite filters from Mozilla’s infrastructure such that they can be served using a standard web server like nginx or apache.
upki-ffi: the C Foreign Function Interface. Built as a cdylib, this crate uses cbindgen to auto-generate a upki.h header file, exposing the revocation query API to C, C++, Go and any other language with C FFI support.
rustls-upki: an integration crate that wires upki’s revocation engine into rustls, enabling any Rust application using rustls to perform CRLite-backed revocation checks transparently.
revoke-test: testing infrastructure for validating revocation queries against known-revoked certificates.

The team recently released v0.1.0, which should help us to gather more feedback on the work we’ve done so far.

How to try it #

If you’d like to try the code in its current form, you’ll need to have a version of the Rust toolchain installed. The easiest way to do this on Ubuntu is using the rustup snap:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# Ensure you have a C compiler in your PATH
sudo apt update
sudo apt install -y build-essential curl

# Install the rustup snap and get the stable toolchain
sudo snap install --classic rustup
rustup install stable

# Install upki
cargo install upki
export PATH="$HOME/.cargo/bin:$PATH"

# Fetch revocation data. This will be done in the background
# when installed through the distro in the future
upki fetch

That should be all you need to install the development version of upki, and you can now use it to run a revocation check by piping certificate output from curl into upki:

1
2


curl -sw '%{certs}' https://google.com | upki revocation check
NotRevoked

Early versions of docs for the C FFI crate and Rust crate documentation are available, but if you’d like to explore, build the project from source, or contribute, the repository is the best place to start. For an example of the C FFI interface in action you can take a look at the upki-go-demo Dirkjan published.

Next Steps #

Now the foundational pieces are in place, our focus is shifting to external consumption, performance, and integration with the wider Linux ecosystem. In the coming days there should be an early 0.1.0 binary release.

We’ll also be doing some performance benchmarking on the initial fetch and of the revocation checks themselves. Currently, each revocation check reads several CRLite filter files into memory. There may be quick wins to improve this, but we’ll benchmark first and see if it warrants optimisation at this time.

We also need to deploy some production infrastructure for serving the CRLite filters. If you follow the steps above, you’ll be fetching from a pre-production web server hosted at https://upki.rustls.dev. We’ve built a Juju charm for operating the CRLite mirror on Kubernetes. This charm packages the upki-mirror binary in a chiselled Rock, and will be deployed into Canonical’s datacentres to serve CRLite data at crlite.ubuntu.com.

Our Ubuntu Foundations team is also working on packaging the various upki components for inclusion in the Ubuntu archive, which will enable you to apt install upki in the future, and also enable us to package and enable it by default in Ubuntu 26.10 and beyond.

Further Down the Road #

While the work above covers what’s immediately in front of us, there is scope to expand upki’s capabilities further. Two areas of interest are Certificate Transparency enforcement, and support for Merkle Tree Certificates.

Certificate Transparency Enforcement #

While upki’s initial focus is on revocation checking, the project also aims to eventually support Certificate Transparency (CT) enforcement. CT is a more modern security measure that relies upon a set of publicly auditable, append-only logs that record every TLS certificate issued by a Certificate Authority (CA). This prevents CAs from issuing fraudulent or erroneous certificates without a means for that fraudulent activity to be discovered - a problem that has bitten organisations in the past.

CT Enforcement would enable clients to refuse to establish a connection unless the server provides cryptographic proof that its certificate has been correctly logged. Browsers like Chrome and Firefox already enforce this, but the rest of the Linux ecosystem would need a tool such as upki to enable such functionality.

Intermediate Preloading #

A correctly configured TLS server should not only send its own certificate, but also the intermediate certificates needed to chain back to a trusted root. In practice, many servers omit the intermediate certificates, and because browsers have quietly worked around this for years, the misconfiguration often goes unnoticed.

Firefox has been preloading all intermediates disclosed to the Common CA Database (CCADB) since Firefox 75, while Chrome and Edge will silently fetch missing intermediates using the Authority Information Access (AIA) extension in the server’s certificate. The result is that a broken certificate chain that works perfectly in every browser will produce an opaque UNKNOWN_ISSUER error when accessed by Linux utilities like curl.

Because upki already maintains a regularly synced local data store, it’s well positioned to ship the known set of intermediates alongside the CRLite filters. This wouldn’t provide a security improvement so much as a usability improvement. It would also bring non-browser clients up to parity with browsers with respect to connection reliability. There is an additional privacy benefit too: rather than fetching a missing intermediate from the issuing CA (which discloses browsing activity to the CA), the intermediate is already present locally.

Merkle Tree Certificates #

Looking even further ahead, upki could support the next generation of web PKI by including support for Merkle Tree Certificates (MTCs). This is an area of active development in the IETF, with Cloudflare and Chrome recently announcing an experimental deployment.

The motivation for MTCs comes largely from the transition to Post-Quantum (PQ) cryptography. PQ signatures are significantly larger than their non-PQ counterparts. The signatures for ML-DSA-44 are 2,420 bytes compared to 64 bytes for ECDSA-P256. A typical TLS handshake today involves multiple signatures and public keys across the certificate chain and CT proofs, which means a simple swap to PQ algorithms would add tens of kilobytes of overhead per connection and likely a noticeable increase in connection latency.

MTCs address this by rethinking how certificates are validated. Rather than transmitting a full certificate chain with multiple signatures, a Certificate Authority can batch certificates into a Merkle Tree and sign only the tree’s root hash. The client then receives just a single signature, a public key, and a compact Merkle tree inclusion proof that demonstrates the certificate’s presence in the batch. The signed tree heads can be distributed to clients out-of-band, meaning the per-handshake overhead is drastically reduced.

Because upki already maintains a local data store that is regularly synced, it could cache tree head data alongside CRLite filters, thereby enabling the inclusion proofs sent during TLS handshakes to be even smaller. Rather than proving inclusion all the way from the leaf to the root, the server could send a “truncated” proof that starts partway up the tree, with the client computing the remainder from data it already has locally. There is a TLS extension being developed to negotiate this.

The implementation of MTCs for TLS is still highly experimental. MTCs are not yet deployed in any browser, but upki will lay the groundwork for Linux system utilities to benefit from this evolution as the technology is adopted.

Summary #

In the few weeks since we announced upki, the core revocation engine has been established and is now functional, the CRLite mirroring tool is working and a production deployment in Canonical’s datacentres is ongoing. We’re now preparing for an alpha release and remain on track for an opt-in preview for Ubuntu 26.04 LTS.

Beyond revocation, we’re keeping a close eye on the evolving PKI landscape and particularly CT enforcement and Merkle Tree Certificates.

I’d like to extend my thanks again to Dirkjan and Joe for their continued collaboration on this work, and the utmost professionalism they’ve demonstrated throughout.

Developing with AI on Ubuntu

Tue, 20 Jan 2026 00:00:00 +0000

This article was originally posted on the Ubuntu Discourse, and is reposted here. I welcome comments and further discussion in that thread.

AI-assisted tooling is becoming more and more common in the workflows of engineers at all experience levels. As I see it, our challenge is one of consideration, enablement and constraint. We must enable those who opt-in to safely and responsibly harness the power of these tools, while respecting those who do not wish to have their platform defined or overwhelmed by this class of software.

The use of AI is a divisive topic among the tech community. I find myself a little in both camps, somewhere between sceptic and advocate. While I’m quick to acknowledge the negative impacts that the use of LLMs can have on open source projects, I’m also surrounded by examples where it has been used responsibly to great effect.

Examples of this include Filippo’s article debugging low-level cryptography with Claude Code, Mitchell’s article on Vibing a Non-Trivial Ghostty Feature, and David’s article How I Program with Agents. These articles come from engineers with proven expertise in careful, precise software engineering, yet they share an important sentiment: AI-assisted tools can be a remarkable force-multiplier when used in conjunction with their lived experience, but care must still be taken to avoid poor outcomes.

The aim of this post is not to convince you to use AI in your work, but rather to introduce the elements of Ubuntu that make it a first-class platform for safe, efficient experimentation and development. My goals for AI and Ubuntu are currently focused on enabling those who want to develop responsibly with AI tools, without negatively impacting the experience of those who’d prefer not to opt-in.

Hardware & Drivers #

AI-specific silicon is moving just as fast as AI software tooling, and without constant work to integrate drivers and userspace tools into Ubuntu, it would be impossible to efficiently utilise this specialised hardware.

Last year we announced that we will ship both NVIDIA’s CUDA and AMD’s ROCm in the Ubuntu archive for Ubuntu 26.04 LTS, in addition to our previous work on OpenVINO. This will make installing the latest drivers and toolkits easier and more secure, with no third-party software repositories. Distributing this software as part of Ubuntu enables us to be proactive in the delivery of security updates and the demonstration of provenance.

Our work is not limited to AMD and NVIDIA; we recently announced support for Qualcomm’s Dragonwing platforms and others. You can read more about our silicon partner projects on our website.

Inference Snaps #

At the Ubuntu Summit 25.10, we released “Inference Snaps” into the wild, which provide a hassle-free mechanism for obtaining the “famous model” you want to work with, but automatically receive a version of that model which is optimised for the silicon in your machine. This removes the need to spend hours on HuggingFace identifying the correct model to download that matches with your hardware, and obviates the need for in-depth understanding of model quantisation and tuning when getting started.

Each of our inference snaps provide a consistent experience: you need only learn the basics once, but can apply those skills to different models as they emerge, whether you’re on a laptop or a server.

At the time of writing, we’ve published beta quality snaps for qwen-vl, deepseek-r1 and gemma3. You can find a current list of snaps in the documentation, along with the silicon-optimised variants.

Sandboxing Agents #

While many start their journey in a web browser chatting to ChatGPT, Claude, Gemini, Perplexity or one of the myriad of alternatives, many developers will find “agentic” tools such as Copilot, Codex, Claude Code or Amp quite attractive. In my experience, agents are a clear level-up in an LLM’s capability for developers, but they can still make poor decisions and are generally safer to run in sandboxed environment at the time of writing.

Where a traditional chat-based AI tool responds reactively to user prompts within a single conversation, an agent operates (semi-)autonomously to pursue goals. It perceives its environment, plans, makes decisions and can call out to external tools and services to achieve those goals. If you grant permission, an agent can read and understand your code, implement features, troubleshoot bugs, optimise performance and many other tasks. The catch is that they often need access to your system - whether that be to modify files or run commands.

Issues such as accidental file deletion, or the inclusion of a spurious (and potentially compromised) dependency are an inevitable failure mode of the current generation of agents due to how they’re trained (see the Reddit post about Claude Code deleting a user’s home directory).

My agent sandboxes itself! #

Some of you will be reading this wondering why additional sandboxing is required, since many of the popular agents advertise their own sandboxing. The fact that some agents include some measures to protect the user’s machine is of course a good thing. The touted benefits include filesystem isolation by restricting the agent to a specific directory, or prompting for approval before modifying files. Some agents also include network sandboxing to restrict network access to a list of approved domains, or by using a custom proxy to impose rules on outbound traffic.

On Linux, these agent-imposed sandboxes are often implemented with bubblewrap, which is “a tool for constructing sandbox environments”, but note that the upstream project’s README includes a section which states that it is not a “ready-made sandbox with a specific security policy”. bubblewrap is a relatively low-level tool that must be given its configuration, which in this case is provided by the agent.

The limitation upon these tools is the shared kernel - a severe kernel exploit could enable an agent to escape from its sandbox. Of course, such vulnerabilities are rare, but note that even if the sandboxing technologies do their job, agents often run in the context of the user’s session, meaning they inherit environment variables which could contain sensitive information. They’re also agent specific: Claude Code’s sandboxing won’t help you if you’re using Cursor or Antigravity.

Depending on your threat model and the project you’re working on, you may deem the built-in sandboxing of coding agents to be sufficient, but there are other options available to Ubuntu users that provide either different, or additional protection…

Sandbox with LXD containers #

Canonical’s LXD works out-of-the-box on Ubuntu, and is a great way to sandbox an agent into a disposable environment where the blast radius is limited should the agent make a mistake. My personal workflow is to create an Ubuntu container (or VM) with my project directory mounted. This way, I can edit my code directly on my filesystem with my preferred (already configured) editor, but have the agent run inside the container.

For example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Initialise the container
lxc init ubuntu:noble dev
# Mount my project directory into the container
lxc config device add -q dev datadir disk source="$HOME/my-project" path=/home/ubuntu/project
# Start the container
lxc start dev
# Get a shell inside the container as the 'ubuntu' user
lxc exec dev -- sudo -u ubuntu -i bash
# Run a command in the container
lxc exec dev -- sudo -u ubuntu -i bash -c "cd project; claude"

You can learn more about LXD in the official documentation and tutorial, as well as specific instructions on enabling GPU data processing in containers/VMs. I’ve written previously about my use of LXD in development.

With LXD, you can choose between running your sandbox as a container or a VM, depending on your project’s needs. If I’m working on a project that requires Kubernetes or similar, I use a VM, but for lighter projects I use system containers, preferring their lower overhead.

Sandbox with LXD VMs #

LXD is best known for its ability to run “system containers”, which are somewhat analogous to Docker/OCI containers, but rather than being focused on a single application (and dependencies), a system container essentially runs an entire Ubuntu user-space (including systemd, etc.). Like OCI containers, however, system containers share the kernel with the host.

In some situations, you may seek more isolation from your host machine by running tools inside a virtual machine with their own kernel. LXD makes this simple - you can follow the same commands as above, but add --vm to the init command:

1
2


# Initialise the virtual machine
lxc init --vm ubuntu:noble dev

You can also configure the virtual machine’s CPU, memory and disk requirements. A simple example is below:

1
2
3
4


lxc init --vm ubuntu:noble dev \
 -c limits.cpu=8 \
 -c limits.memory=8GiB \
 -d root,size=100GiB

You can find more details on instance configuration in the LXD documentation.

Sandbox with Multipass #

Multipass provides on-demand access to Ubuntu VMs from any workstation - whether that workstation is running Linux, macOS or Windows. It is designed to replicate, in a lightweight way, the experience of provisioning a simple Ubuntu VM on a cloud.

Multipass’ scope is more limited than LXD, but for many users it provides a simple on-ramp for development with Ubuntu. Where it lacks advanced features like GPU passthrough, it boasts a simplified CLI and a first-class GUI client.

To get started similarly to the LXD example above, try the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Install Multipass
sudo snap install multipass
# Launch an instance
multipass launch noble -n dev
# Mount your project directory
multipass mount ~/my-project dev:/home/ubuntu/project
# Get a shell in the instance
multipass shell dev
# Run a command in the instance
multipass exec dev -- claude

You can find more details on how to configure and manage instances in the docs.

Sandbox with WSL #

If you’re on Windows, development with WSL includes first-class support for GPU acceleration, and is even supported for use with the NVIDIA AI Workbench, NVIDIA NIM and CUDA.

Ubuntu is the default Linux distribution for WSL, and you can find more information about how to set up and use Ubuntu on WSL in our documentation. WSL benefits from all the same technologies as a “regular” Ubuntu install, including the ability to use Snaps, Docker and LXD.

For the enterprise developer, we recently announced Ubuntu Pro for WSL, as well as the ability to manage WSL instances using Landscape, making it easier to get access to first-class developer tooling with Ubuntu on your corporate machine.

Summary #

While opinion remains divided on the value and impact of current AI tooling, its presence in modern development workflows and its demands on underlying compute infrastructure are difficult to ignore.

Developers who wish to experiment need reliable access to modern hardware, predictable tooling, and strong isolation boundaries. Ubuntu’s role is not to dictate how these tools are used, but to provide a stable and dependable platform on which they can be explored and deployed safely, without compromising security, provenance, or the day-to-day experience of those who choose to opt out.

In addition to powering development workflows, Ubuntu makes for a dependable production operating system for your workloads. We’re building Canonical Kubernetes with first-class GPU support, Kubeflow and MLFlow for model training and serving and a suite of applications like PostgreSQL, MySQL, Opensearch, as well as other data-centric tools such as Kafka and Spark that can be deployed with full Ubuntu Pro support. Let me know if you’d find value in a follow-up post on those topics!

Addressing Linux's Missing PKI Infrastructure

Mon, 08 Dec 2025 00:00:00 +0000

This article was originally posted on the Ubuntu Discourse, and is reposted here. I welcome comments and further discussion in that thread.

Earlier this year, LWN featured an excellent article titled “Linux’s missing CRL infrastructure”. The article highlighted a number of key issues surrounding traditional Public Key Infrastructure (PKI), but critically noted how even the available measures are effectively ignored by the majority of system-level software on Linux.

One of the motivators for the discussion is that the Online Certificate Status Protocol (OCSP) will cease to be supported by Let’s Encrypt. The remaining alternative is to use Certificate Revocation Lists (CRLs), yet there is little or no support for managing (or even querying) these lists in most Linux system utilities.

To solve this, I’m happy to share that in partnership with rustls maintainers Dirkjan Ochtman and Joe Birr-Pixton, we’re starting the development of upki: a universal PKI tool. This project initially aims to close the revocation gap through the combination of a new system utility and eventual library support for common TLS/SSL libraries such as OpenSSL, GnuTLS and rustls.

The Problem #

Online Certificate Authorities responsible for issuing TLS certificates have long had mechanisms for revoking known bad certificates. What constitutes a known bad certificate varies, but generally it means a certificate was issued either in error, or by a malicious actor of some form. There have been two primary mechanisms for this revocation: Certificate Revocation Lists (CRLs) and the Online Certificate Status Protocol (OCSP).

In July 2024, Let’s Encrypt announced the deprecation of support for the Online Certificate Status Protocol (OCSP). This wasn’t entirely unexpected - the protocol has suffered from privacy defects which leak the browsing habits of users to Certificate Authorities. Various implementations have also suffered reliability issues that forced most implementers to adopt “soft-fail” policies, rendering the checks largely ineffective.

The deprecation of OCSP leaves us with CRLs. Both Windows and macOS rely on operating system components to centralise the fetching and parsing of CRLs, but Linux has traditionally delegated this responsibility to individual applications. This is done most effectively in browsers such as Mozilla Firefox, Google Chrome and Chromium, but this has been achieved with bespoke infrastructure.

However, Linux itself has fallen short by not providing consistent revocation checking infrastructure for the rest of userspace - tools such as curl, system package managers and language runtimes lack a unified mechanism to process this data.

The ideal solution to this problem, which is slowly becoming more prevalent, is to issue short-lived credentials with an expiration of 10 days or less, somewhat removing the need for complicated revocation infrastructure, but reducing certificate lifetimes is happening slowly and requires significant automation.

CRLite #

There are several key challenges with CRLs in practice - the size of the list has grown dramatically as the web has scaled, and one must collate CRLs from all relevant certificate authorities in order to be useful. CRLite was originally proposed by researchers at IEEE S&P and subsequently adopted in Mozilla Firefox. It offers a pragmatic solution to the problem of distributing large CRL datasets to client machines.

In a recent blog post, Mozilla outlined how their CRLite implementation meant that on average users “downloaded 300kB of revocation data per day, a 4MB snapshot every 45 days and a sequence of “delta-updates” in-between”, which amounts to CRLite being 1000x more bandwidth-efficient than daily CRL downloads.

At its core, CRLite is a data structure compressing the full set of web-PKI revocations into a compact, efficiently queryable form. You can find more information about CRLite’s design and implementation on Mozilla’s Security Blog.

Introducing upki #

Following our work on oxidizing Ubuntu, Dirkjan reached out to me with a proposal to introduce a system-level utility backed by CRLite to non-browser users.

upki will be an open source project, initially packaged for Ubuntu but available to all Linux distributions, and likely portable to other Unix-like operating systems. Written in Rust, upki supports three roles:

Server-side mirroring tool: responsible for downloading and mirroring the CRLite filters provided by Mozilla, enabling us to operate independent CDN infrastructure for CRLite users, and serving them to clients. This will insulate upki from changes in the Mozilla backend, and enable standing up an independent data source if required. The server-side tool will manifest as a service that periodically checks the Mozilla Firefox CRLite filters, downloads and validates the files, and serves them.
Client-side sync tool: run regularly by a systemd-timer, network-up events or similar, this tool ensures the contents of the CDN are reflected in the on-disk filter cache. This will be extremely low on bandwidth and CPU usage assuming everything is up to date.
Client-side query tool: a CLI interface for querying revocation data. This will be useful for monitoring and deployment workflows, as well as for users without a good C FFI.

The latter two roles are served by a single Rust binary that runs in different modes depending on how it is invoked. The server-side tool will be a separate binary, since its use will be much less widespread. Under the hood, all of this will be powered by Rust library crates that can be integrated in other projects via crates.io.

For the initial release, Canonical will stand up the backend infrastructure required to mirror and serve the CRLite data for upki users, though the backend will be configurable. This prevents unbounded load on Mozilla’s infrastructure and ensures long-term stability even if Firefox’s internal formats evolve.

Ecosystem Compatibility #

So far we’ve covered the introduction of a new Rust binary (and crate) for supporting the fetching, serving and querying of CRL data, but that doesn’t provide much service to the existing ecosystem of Linux applications and libraries in the problem statement.

The upki project will also provide a shared object library for a stable ABI that allows C and C-FFI programs to make revocation queries, using the contents of the on-disk filter cache.

Once upki is released and available, work can begin on integrating existing crypto libraries such as OpenSSL, GNUtls and rustls. This will be performed through the shared object library by means of an optional callback mechanism these libraries can use to check the revocation lists before establishing a connection to a given server with a certificate.

Timeline #

While we’ve been discussing this project for a couple of months, ironing out the details of funding and design, work will soon begin on the initial implementation of upki.

Our aim is to make upki available as an opt-in preview for the release of Ubuntu 26.04 LTS, meaning we’ll need to complete the implementation of the server/client functionality, and bootstrap the mirroring/serving infrastructure at Canonical before April 2026.

In the following Ubuntu release cycle, the run up to Ubuntu 26.10, we’ll aim to ship the tool by default on Ubuntu systems, and begin work on integration with the likes of NSS, OpenSSL, GNUtls and rustls.

Summary #

Linux has a clear gap in its handling of revocation data for PKIs. Over the coming months we’re hoping to address that gap by developing upki not just for Ubuntu, but for the entire ecosystem. Thanks to Mozilla’s work on CRLite, and the expertise of Dirkjan and Joe, we’re confident that we’ll deliver a resilient and efficient solution that should make a meaningful contribution to systems security across the web.

If you’d like to do more reading on the subject, I’d recommend the following:

LWN.net: Linux’s missing CRL infrastructure
Mozilla Security Blog: CRLite Part 1: All Web PKI Revocations Compressed
Mozilla Security Blog: CRLite Part 2: End-to-End Design
Let’s Encrypt: Replacing OCSP with CRLs
IEEE Symposium on Security & Privacy: CRLite: A Scalable System for Pushing All TLS Revocations to All Browsers

Secure Boot & TPM-backed Full Disk Encryption on NixOS

Sat, 20 Apr 2024 00:00:00 +0000

Update: Since I wrote this post, I have been made aware of a particular situation where, at the time I write this (2025-01-17), the steps described in this article will result in a setup that is still (in many cases) vulnerable to an attack where the attacker has physical access to the machine. This may be acceptable in your threat model, but I’d encourage you to read the excellent article to gain a full understanding of the issue.

Introduction #

For the last decade (whoa…) or so, I’ve defaulted to using LUKS-encrypted drives for my machines. In general, I configure an unencrypted boot/EFI partition, then place either an ext4 or btrfs filesystem inside a LUKS container which is used for the root partition.

Some of my machines also have extra disks: my desktop has a 1TB NVMe drive for root, and a 2TB NVMe “data” drive mounted in my home directory under /home/jon/data. I don’t like having to type two different encryption passphrases at boot, so I usually have the extra disk automatically unlocked by putting the key in a file on the root drive, and placing an entry in /etc/crypttab.

This setup works fine for desktop machines, but it’s cumbersome on headless machines because unattended reboots require the disk passphrase to be entered at boot. Even then, all of my computers are exclusively used by me, and this setup means I have to enter two passwords on every boot to get to a working desktop environment (one for the disk, and one for the login manager).

I solved this recently with a combination of Secure Boot and a Trusted Platform Module (TPM), so let’s look at those first with a brief and high-level overview of each.

What’s a TPM? #

Most machines that have been manufactured in the last decade, and certainly in the last 5 years, contain a cryptographic coprocessor conforming to the Trusted Platform Module (TPM) spec. The TPM is a dedicated microcontroller primarily used for verifying the integrity of a machine.

TPMs can be used for storing cryptographic key material and performing basic cryptographic operations. The general premise is that keys can be loaded into the TPM, which enables the TPM to perform cryptographic operations using that key (signing, encrypting, etc.), but the key cannot be recovered or read from the TPM unless certain conditions are met. TPMs also provide other facilities such as secure random number generation, which in turn enables them to securely generate cryptographic keys.

Verifying system integrity essentially boils down to being able to ensure the machine hasn’t been tampered with between boots, and that the boot process itself hasn’t been compromised. A given firmware or operating system can take hardware “measurements” and store those measurements in dedicated slots called Platform Configuration Registers (PCRs). The measurements pertain to the underlying hardware and configuration of the machine. The TPM itself never performs the actual verification of the PCRs, and in fact has no knowledge of whether a measurement is inherently “good” or “bad”, but it can provide signed attestations of their values, which are then judged by the application requesting the attestation according to some policy.

Each PCR contains a hash representing a particular hardware measurement, which can be read at any time, but cannot be overwritten. Rather than allowing a traditional write operation, PCRs are updated through an “extend” operation which depends on the previous hash value, creating a chain of trust not dissimilar from how a blockchain is formed. This means that a given measurement can never be fully removed from the TPM.

Once the measurements are stored in the PCRs, there are various times and purposes for which the firmware or an operating system might read them - one example is for remote attestation during login to a system. In this scenario the attestation can be used to verify that the machine hasn’t been tampered with (perhaps in an Evil Maid attack). One could also store a Certificate Authority signing key in a TPM, and have the Certificate Authority software interface with the TPM to sign certificates using the PKCS#11 standard.

My use-case is to enabling the TPM to provide the passphrase to unlock a LUKS-encrypted disk, which is what I’ll focus on in this post.

Secure Boot #

Secure Boot is the mechanism by which the code executed by a machine’s Unified Extensible Firmware Interface (UEFI) can be verified as trusted. In the vast majority of cases, the first thing executed by the UEFI is a bootloader.

When Secure Boot is enabled, each binary executed by the UEFI must contain a checksum and a signature - which the UEFI verifies before launching the code. In the case that either the checksum or signature do not match, the UEFI will refuse the execute the code, and the boot process will halt.

Many OEM machines ship with Microsoft Windows installed, and thus ship with the necessary keys to validate signatures created with Microsoft’s certificate authority. Linux systems are able to utilise these keys through shim - a small and easily verifiable piece of software which is signed by Microsoft. shim sits between the UEFI and the bootloader in the boot process, obviating the need for every Linux bootloader to be signed by Microsoft on every release. The shim is designed to extend trust from the keys trusted by the computer’s firmware to a new set of keys controlled by the operating system.

But what does Secure Boot get us in reality? By signing the kernel, and in some cases a single UEFI PE binary known as a Unified Kernel Image (UKI) (which contains the bootloader, the kernel, the command-line used to boot the kernel, and other resources), one can be reasonably sure that the boot process hasn’t been tampered with.

This process thwarts a number of common physical attack vectors, such as manipulating the kernel command line to bypass the machine’s login and drop straight to a root shell - and combined with disk encryption can prevent offline data transfer from the machine. It also defends against malware which compromises the operating system’s boot process such that it can start before the OS and obfuscate it’s presence.

Threat Modelling #

As with any security measure, Secure Boot is not a silver bullet. You should always consider your own personal threat model, and the sorts of attacks you’re looking to defend against.

For example, Secure Boot can help prevent the sort of malware infections described in the previous section, but if an attacker gets physical access to your machine, and the UEFI isn’t adequately protected, they could simply disable secure boot and carry on unhindered.

I mitigate this by password protecting the UEFI. This isn’t perfect, but is likely sufficient protection for my threat model which is more about protecting chancers and petty thieves from gaining access to my information, than from determined attackers who gain physical access to my property.

Storing keys in a TPM is theoretically safe, in that each TPM has a unique seed which cannot be retrieved, and enables the TPM to deterministically generate keys between reboots. It’s very difficult to retrieve the seed, and thus very difficult to duplicate a TPM, but not impossible. Even then, the Linux kernel’s communication with the TPM on-the-wire is unencrypted, and the same can be said for many other subsystems which use the TPM. A recent example of this vulnerability was demonstrated by sniffing a Bitlocker key off the LPC bus in a Lenovo X1 Carbon laptop (using a Raspberry Pi Pico, no less). In many modern machines this is mitigated by the TPM being on-CPU, but the point still stands.

I choose to enroll Microsoft’s platform keys, which in theory degrades the security of my device in the case that Microsoft’s signing key is compromised, though all of my machines are compatible with fwupd and can receive updates to the database through that mechanism if required (and in fact have done in the past 18 months). This could be further mitigated by using custom keys and certificates for the full chain, but this is more overhead for daily operations and updates. It’s also worth considering whether you have the resources to fully secure your own chain - especially by comparison to Microsoft who spend tens of millions of dollars per year on security. If an attacker wants your information, and are able to compromise Microsoft’s CA, your own CA may not be such a hurdle.

Security measures are always a trade-off between Confidentiality, Availability and Integrity (CIA). In general, the more rigidly secure boot is implemented and configured, the more you’re protecting confidentiality and integrity. The choices I’ve made are slightly more in favour of availability, but nonetheless raise the bar for any attacker significantly.

Enabling Secure Boot on NixOS #

Now for the fun part! The process for enabling Secure Boot on NixOS has simplified in recent months owing to the creation of lanzaboote - a project which takes of preparing and signing Unified Kernel Images containing a custom stub, the bootloader, the Linux kernel, the kernel’s initrd and the kernel command line. lanzaboote also takes care of installing the UKI on the ESP partition so the UEFI can execute it at boot.

The lanzaboote stub differs slightly from systemd-stub, in that it doesn’t require the kernel and initrd to be part of the UKI. This is important for a generation-based operating system like NixOS because bundling the kernel and initrd into a new UKI for every generation would consume a lot of disk space, and quickly exhaust the ESP on most machines. In lanzaboote’s implementation, the kernel and initrd are stored separately on the ESP, and the chain of trust is preserved by validating the signature of the kernel, and embedding a cryptographic hash of the initrd into the signed UKI.

The project takes advantage of systems that have bootspec enabled, which is a relatively recent NixOS RFC that ensures configured machines maintain a file containing a set of memoised facts about a system’s closure. Bootspec aims to “provide more uniform feature support” to bootloaders in the NixOS ecosystem and “enable NixOS users to implement custom bootloader tools and policy” - of which lanzaboote is one.

Generating Secure Boot Keys #

The first step is to generate some keys for the secure boot process. This can be achieved using the sbctl package:

1
2
3
4


❯ sudo nix run nixpkgs#sbctl create-keys
Created Owner UUID 6ac34cc3-a23d-9745-ef33-a03f523d20a3
Creating secure boot keys...✓
Secure boot keys created!

This should only take a few seconds at maximum, and will result in a set of keys being populated in /etc/secureboot:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


❯ tree /etc/secureboot
/etc/secureboot
├── files.db
├── GUID
└── keys
 ├── db
 │ ├── db.key
 │ └── db.pem
 ├── dbx
 │ ├── dbx.key
 │ └── dbx.pem
 ├── KEK
 │ ├── KEK.key
 │ └── KEK.pem
 └── PK
 ├── PK.key
 └── PK.pem

Enable Bootspec #

Ensure that bootspec is enabled in your Nix configuration. You can see this in my flake for my desktop machine on Github:

1
2
3


{
 boot.bootspec.enabled = true;
}

Enable `lanzaboote` #

I use a flake to configure all of my machines, so I’m able to get access to lanzaboote by adding the upstream flake as an input to my own nixos-config flake:

1
2
3
4
5


# ...
inputs = {
 lanzaboote.url = "github:nix-community/lanzaboote";
};
# ...

Once you’ve added lanzaboote as a dependency, you’ll need to import the lanzaboote module:

1
2
3


# ...
imports = [ lanzaboote.nixosModules.lanzaboote ];
# ...

In my flake, I use a custom helper function to build NixOS configurations, so the module is passed directly to lib.nixosSystem through the modules attribute.

The lanzaboote module replaces the systemd-boot module, and as such you must explicitly disable systemd-boot when enabling lanzaboote. Additionally, if you wish to use the TPM for disk unlock (described in the next section), you must use the systemd initrd hooks (or something like clevis):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


boot = {
 initrd.systemd.enable = true;

 loader.systemd-boot.enable = lib.mkForce false;

 lanzaboote = {
 enable = true;
 pkiBundle = "/etc/secureboot";
 };
};

This is represented in my config here.

Once enabled, rebuild your system (in my case with sudo nixos-rebuild switch --flake /home/jon/nixos-config) and verify that your machine is ready for Secure Boot. Don’t panic about the kernel images being reported as not signed, this is expected:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


❯ sudo nix run unstable#sbctl verify
Verifying file database and EFI images in /boot...
✓ /boot/EFI/BOOT/BOOTX64.EFI is signed
✓ /boot/EFI/Linux/nixos-generation-414-376jna572gsb23snqs67t7s4bwxzb3epblmdnzweghuepopml2va.efi is signed
✓ /boot/EFI/Linux/nixos-generation-415-iqulgohymbdppgtxzho6ou3fcuxjbxhumpzm4vojmipwy3sbmuna.efi is signed
✓ /boot/EFI/Linux/nixos-generation-416-kxnzioafnduwwck3oypo7rqwtoat745czp2bpehoufp4yqiawypa.efi is signed
✗ /boot/EFI/nixos/kernel-6.8.2-242idodyvf36cpl6s5dskjy6mo4tjhszuwa3hye7qcjyuo5vnehq.efi is not signed
✗ /boot/EFI/nixos/kernel-6.8.5-zqulrwsucm6okcyns6v2jhh6fregk3bvsdth3yloqfymfbgnh64a.efi is not signed
✗ /boot/EFI/nixos/kernel-6.8.7-6mmixkr6ewywm5swgbi5ethbpgnyia4borzmkevcjx7n7t3mtida.efi is not signed
✓ /boot/EFI/systemd/systemd-bootx64.efi is signed

Prepare the UEFI #

Reboot your machine and enter the UEFI interface. This part of the process will vary from machine to machine depending on the UEFI implementation, but you’re looking to enable Secure Boot, and clear the preloaded Secure Boot keys. This may be referred to as “Setup Mode”, or erasing the “Platform Keys”.

While you’re here, I’d also advise setting a UEFI password before rebooting back into NixOS.

Enroll Secure Boot Keys #

The final stage in the process is to enroll your newly generated Secure Boot keys from step 1 into the UEFI. This is again achieved with sbctl:

1
2
3
4


❯ sudo nix run nixpkgs#sbctl enroll-keys -- --microsoft
Enrolling keys to EFI variables...
With vendor keys from microsoft...✓
Enrolled keys to the EFI variables!

I chose to use the --microsoft option to also enroll the UEFI vendor certificates from Microsoft. Some systems contain firmware that is signed and validated when Secure Boot is enabled, and omitting the Microsoft keys could prevent your device from booting - omit this option with caution!

Verify Secure Boot #

Once you’ve enrolled the keys, reboot the machine back into NixOS and use bootctl to confirm that Secure Boot is in fact enabled:

1
2
3
4
5
6
7


❯ bootctl status
System:
 Firmware: UEFI 2.80 (American Megatrends 5.26)
 Firmware Arch: x64
 Secure Boot: enabled (user)
 TPM2 Support: yes
 Boot into FW: supported

TPM Unlock of Root Partition #

Now that we’re (reasonably) confident that no one can tamper with the boot process, we can progress to allowing the machine to auto-unlock the encrypted disk using a key stored in the TPM.

This is actually more common than you might think - Windows has enabled this behaviour by default for some time with Bitlocker disk encryption, and Canonical is also working on bringing TPM-backed full disk encryption to Ubuntu.

This is probably the easiest step of them all! A simple invocation of systemd-cryptenroll is all that’s required. The arguments below instruct the machine that PCRs 0, 2, 7 and 12 should be measured and verified before the TPM is allowed to unlock the disk.

According to the Linux TPM PCR Regsitry, this means the following are measured before the LUKS key is presented:

PCR 0: Core system firmware executable code
PCR 2: Extended or pluggable executable code
PCR 7: SecureBoot state
PCR 12: Kernel command line, system credentials and system configuration images

1

❯ sudo systemd-cryptenroll --tpm2-device=auto --tpm2-pcrs=0+2+7+12 --wipe-slot=tpm2 /dev/nvme0n1p2

And that’s it! The next time you reboot, your disk should be automatically unlocked by the TPM, and your machine should boot straight to your display manager, or the TTY login if no display manager is configured.

Useful Resources #

None of the knowledge in this post is novel, but rather the culmination of some knowledge acquired over the past few years, and some more targeted reading more recently. In the process, I learned a bunch from the following:

Summary #

About 5 years ago, I was sporting a fully secure-boot enabled Dell XPS 13 running Arch Linux. Back then, the process was complicated, manual, and required a lot of maintenance between upgrades. For me, it was more pain than gain, but an interesting learning experience nonetheless.

When I sat down earlier this year to enable Secure Boot on NixOS, I’d set aside a few hours. I was astounded that 10 minutes later I was finished. I wrote this post as a memo to my future self, but also to illustrate how simple it can be to enable Secure Boot and TPM disk unlock in 2024.

I don’t claim to be an expert on the inner workings of TPMs, nor Secure Boot. The things I can say for certain are that TPMs are complex, that there are improvements that could be made to Linux’s interactions with the TPM, and that a determined and well-resourced attacker is likely going to succeed one way or another.

If you spot an inaccuracy in this post, reach out and let me know on Mastodon, on Telegram, by email, or however you prefer!

Until next time!

Update 2024/04/29: Thanks to @pimeys for pointing out that one must enable the systemd initrd hooks systemd-cryptenroll to function correctly, and also that PCR 12 must be measured to prevent the LUKS key from being released if the kernel command line has been modified.

Zero-Trust SSH on Microsoft Azure

Mon, 12 Oct 2020 00:00:00 +0000

This post was originally posted on LinkedIn on 12 October 2020. I’ve posted it to my blog retrospectively, but the article is unchanged. SSHizzle was released as open source software back in 2020, but has not received any maintenance since I left Thales. I do not recommend deploying SSHizzle, but the article will hopefully remain interesting for the background and principles behind the project.

Introduction #

Since its introduction, Secure Shell (ssh) has become the defacto solution for remotely managing Linux and Unix based systems, and has become only more prolific with the increasing popularity of the cloud. I can’t think of a cloud provider that supplies access to Linux machines for whom SSH is not the default, and likely most active way for their customers to manage their resources. The same applies in large enterprise networks, with OpenSSH Server and Client packages even available in Windows 10 and Windows Server 2019 in more recent releases.

A key challenge with SSH is managing the authentication of trusted users at scale. OpenSSH supports three options by default, including password authentication, key authentication and certificate authentication. All of these methods pose varying challenges both to operations and security.

Hello, SSHizzle #

Today, I’m delighted to release SSHizzle on behalf of Thales Secure Communications & Information Systems UK, a tool designed to illustrate the simplified management of SSH users and authentication both for administrators and users using commodity cloud resources. SSHizzle was inspired by the work of Jeremy Stott (@stoggi) and the sshrimp project which implements a similar solution for AWS.

SSHizzle strives to be as simple to configure, use and run as possible. It benefits from tight integration with Microsoft Azure Active Directory, Azure Key Vault and works seamlessly with almost all existing workflows. SShizzle simplifies the adoption and management of SSH Certificates.

In summary, SSHizzle aims to:

Reduce the complexity of on-boarding new users into environments
Provide a good user experience to engineers and developers
Increase the observability of SSH credentials
Eliminate the need to distribute many public keys in production
Reduce the need to manage the lifecycle of SSH keys in production
Be compatible with existing SSH-based workflows
Require minimal configuration of hosts and servers

SSHizzle comprises of two key components:

A serverless SSH Certificate Authority (CA)
An SSH Agent for client machines

SSHizzle’s Serverless Certificate Authority #

SSHizzle’s SSH Certificate Authority (CA) stores its key in an Azure Key Vault and operates from a serverless function operated by Azure Functions. Azure Key Vault is a managed service provided by Microsoft that enables the secure storage and management of secrets, keys and certificates. In the case of SSHizzle, the Azure Key Vault stores the private key for the CA.

A serverless function is, generally, a programmatic function whose execution is triggered and run ephemerally on managed infrastructure. This means there is no dedicated server consistently running or listening for connections as with traditional server deployments. In the case of SSHizzle, the serverless function is triggered by an HTTP listener, which when called by an authenticated user triggers the Azure Functions backend to run the function on any available, suitable infrastructure. Once the function has run and returned, the context that executed the function (perhaps a container or a VM in the underlying implementation) is destroyed. This has numerous benefits including more granular billing (serverless functions are often billed by the minute for execution), but also a reduced attack surface as the authentication and frontend to the function are all controlled by Microsoft Azure.

SSHizzle’s backend function can only be invoked by a user authenticated against its parent Azure Active Directory tenant. Once a user has authenticated, it takes a randomly generated SSH public key, signs it using the key stored in the Azure Key Vault, and returns an SSH certificate to the agent that invoked the function.

The SSH certificate has a number of key advantages over traditional SSH keys:

Expiry Time: SSHizzle only issues credentials valid for 2 minutes, eliminating the need to manage the lifecycle of SSH user credentials.
Access Limitations: SSHizzle will issue credentials that are only valid for use from the IP address that requested them. Should the credential be intercepted on the way back to the agent, it will be invalid for use unless the connection is from the original IP.
Forced Commands: SSHizzle does not yet implement a UI to manage this feature but with minor modifications, SSH certificates can be issued that are only valid for specific command executions. This allows the issuance of single-purpose, scope limited credentials; essentially a light form of Privileged Access Management.
Feature Restriction: SSH certificates have the ability to enable or disable certain SSH session functionality such as Agent Forwarding, Port Forwarding, X11 Forwarding etc.
Observable ID: The ID of each certificate (which appears in auth logs and system logs) contains key information to track how and when each credential was issued - making forensic investigation easier.

SSHizzle’s SSH Agent #

An SSH agent performs the function of a key manager for SSH. In a typical setup, a user might store their SSH keys in their home directory. The SSH agent is a background process that is invoked when the user attempts to SSH into a remote host. Its job is to sign challenges with the user’s private keys in order to authenticate with a remote host. An SSH Agent cannot perform any function other than signing messages; it does not write key material to disk and it does not permit the export of private keys.

Where the default SSH Agent reads its private keys from the user’s home directory, SSHizzle’s agent behaves quite differently because it needs to have a public key signed by the key stored in the Azure Key Vault. If a user’s SSH config invokes the SSHizzle Agent to authenticate with a host, the following process occurs:

The agent generates a new SSH public/private key pair.
The agent checks if it is authenticated with Azure Active Directory. If required, the agent opens a browser and asks the user to authenticate.
The agent invokes the serverless function, passing it the generated public key.
The serverless function signs the public key and returns an SSH certificate to the agent.
The agent uses the newly acquired certificate to authenticate with the host.

Benefits #

Using this approach, SSHizzle is able to meet its goals:

Onboarding complexity is reduced; irrespective of the class of user (administrator, developer, analyst, etc.), if a user exists in the Azure Active Directory tenant, and possess the relevant roles, they can authenticate using SSHizzle.

The user experience is slick; users are prompted to authenticate with Azure AD in the same way they are to interact with Microsoft 365 and any other services available through the tenant. Additionally, they can use the same frictionless multi-factor authentication methods such as the Microsoft Authenticator app with no additional setup. This not only provides a consistent UX, but increases the security of the solution.

Credential issuance is more observable; the certificate IDs are verbose and easy to identify in system logs. There is detailed audit information available about specific serverless function invocations through the Azure Portal.

There is no need to distribute many keys to SSH hosts; servers must simply be configured to trust a single user certificate authority.

Credential lifecycle management complexity is reduced; because credentials are only valid for 2 minutes by default, there is no need to revoke credentials. There is little chance that a valid SSH certificate will leak. Off-boarding users is as simple as disabling/removing their account in Azure Active Directory.

The SSHizzle Agent is compatible with existing workflows; provided the user’s local SSH config is configured such that all hosts requiring SSHizzle authentication invoke the agent, then other applications like scp, rsync and Visual Studio Code Remote Development work seamlessly. This configuration is particularly simple when combined with wildcard host matching - i.e. configure all SSH connections to *.your-corp-domain.org to invoke the agent.

SSHizzle is trivial to configure; servers require the inclusion of a single CA public key file, and single line of config in the sshd_config file. On client machines, the SSHizzle Agent is a single binary that can be started in the background.

SSHizzle is extremely cost effective to deploy and run. Azure Functions are free for the first 1,000,000 invocations per month, and the corresponding storage and key vault operations aren’t likely to total more than a a few tens of dollars per month with moderate usage.

Limitations and Possible Extensions #

As previously alluded to, the released version of SSHizzle is a proof-of-concept. It demonstrates the concept of utilising a serverless function performing the role of an SSH Certificate Authority, combined with a strong identity provider to simplify the adoption and management of SSH Certificates across an enterprise.

Currently, it lacks a few features one might consider useful for widespread adoption:

Per-user policy for ForceCommands and session features (port forwarding, etc.)

Ability to check if the user authenticating should be interacting with the specified host over SSH, currently if the user is valid in the tenant, a certificate is issued. Their actual access to the end host is governed by whether or not an account exists with their username on the server.

Provisioning of a Host Certificate Authority to avoid host key warnings and prompts

Exercise caution before deploying into production, but do experiment, and perhaps even submit a pull request…

Try it out! #

The code for SSHizzle is available on Github, complete with Terraform automation to deploy demonstration resources on Microsoft Azure.

Security on Jon Seager

ntpd-rs: it's about time!

Introduction #

NTP, NTS, and PTP #

Proven at Scale #

A Single, Memory-Safe Utility for NTP and PTP #

Timelines and Goals #

About the Trifecta Tech Foundation #

Summary #

An update on upki

Architecture & Progress #

How to try it #

Next Steps #

Further Down the Road #

Certificate Transparency Enforcement #

Intermediate Preloading #

Merkle Tree Certificates #

Summary #

Developing with AI on Ubuntu

Hardware & Drivers #

Inference Snaps #

Sandboxing Agents #

My agent sandboxes itself! #

Sandbox with LXD containers #

Sandbox with LXD VMs #

Sandbox with Multipass #

Sandbox with WSL #

Summary #

Addressing Linux's Missing PKI Infrastructure

The Problem #

CRLite #

Introducing upki #

Ecosystem Compatibility #

Timeline #

Summary #

Secure Boot & TPM-backed Full Disk Encryption on NixOS

Introduction #

What’s a TPM? #

Secure Boot #

Threat Modelling #

Enabling Secure Boot on NixOS #

Generating Secure Boot Keys #

Enable Bootspec #

Enable lanzaboote #

Prepare the UEFI #

Enroll Secure Boot Keys #

Verify Secure Boot #

TPM Unlock of Root Partition #

Useful Resources #

Summary #

Zero-Trust SSH on Microsoft Azure

Introduction #

Hello, SSHizzle #

SSHizzle’s Serverless Certificate Authority #

SSHizzle’s SSH Agent #

Benefits #

Limitations and Possible Extensions #

Try it out! #

Enable `lanzaboote` #