Snapcraft on Jon Seager

Crafting Your Software

Mon, 21 Jul 2025 00:00:00 +0000

This article was originally posted on the Ubuntu Discourse, and is reposted here. I welcome comments and further discussion in that thread.

Packaging software is notoriously tricky. Every language, framework, and build system has its quirks, and the variety of artifact types — from Debian packages to OCI images and cloud images — only adds to the complexity.

Over the past decade, Canonical has been refining a family of tools called “crafts” to tame this complexity and make building, testing, and releasing software across ecosystems much simpler.

The journey began on 23rd June 2015 when the first commit was made to Snapcraft, the tool used to build Snap packages. For years, Snapcraft was the only craft in our portfolio, but in the last five years, we’ve generalized much of what we learned about building, testing, and releasing software into a number of “crafts” for building different artifact types.

Last month, I outlined Canonical’s plan to build debcraft as a next-generation way to build Debian packages. In this post I’ll talk about what exactly makes a craft, and why you should bother learning to use them.

Software build lifecycle #

At the heart of all our crafts is craft-parts, which according to the documentation “provides a mechanism to obtain data from different sources, process it in various ways, and prepare a filesystem sub-tree suitable for packaging”.

Put simply, craft-parts gives developers consistent tools to fetch, build, and prepare software from any ecosystem for packaging into various formats.

Lifecycle stages #

Every part has a minimum of four lifecycle stages:

PULL: source code or binary artifacts, along with dependencies are pulled from various sources
BUILD: software is built automatically by a plugin, or a set of custom steps defined by the developer
STAGE: select outputs from the BUILD phase are copied to a unified staging area for all parts
PRIME: files from the staging area are copied to the priming area for use in the final artifact.

The STAGE and PRIME steps are similar, except that PRIME only happens after all parts of the build are staged. Additionally, STAGE provides the opportunity for parts to build/supply dependencies for other parts, but that might not be required in the final artifact.

Lifecycle in the CLI #

The lifecycle stages aren’t just in the build recipe, they’re also first-class citizens in each craft’s CLI, thanks to the craft-cli library. This ensures a consistent command-line experience across all craft tools.

Take the following examples:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Run the full process including PULL, BUILD, STAGE, PRIME and then pack the final artifact
snapcraft pack
charmcraft pack
rockcraft pack

# Run the process up to the end of the STAGE step
rockcraft stage

# Run the process up to the PRIME step
charmcraft prime

This design feature supports a smoother iterative development and debugging workflow for building and testing software artifacts.

Part definition #

The parts of a build vary in complexity - some require two-three trivial lines, others require detailed specification of dependencies, build flags, environment variables and steps. The best way to understand the flexibility of this system is by looking at some examples.

First, consider this (annotated) example from my icloudpd snap:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


icloudpd:
 # Use the 'python' plugin to build the
 # software. This takes care of identifying
 # Python package dependencies, building the wheel
 # and ensuring the project's dependencies are staged
 # appropriately.
 plugin: python
 # Fetch the project from Github, using the tag the matches
 # the version of the project.
 source: https://github.com/icloud-photos-downloader/icloud_photos_downloader
 source-tag: v$SNAPCRAFT_PROJECT_VERSION
 source-type: git

This spec is everything required to fetch, build and stage the important bits required to run the software - in this case a Python wheel and its dependencies.

Some projects might require more set up, perhaps an additional package is required or a specific version of a dependency is needed. Let’s take a look at a slightly more complex example taken from my zinc-k8s-operator project:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


kube-log-runner:
 # Use the 'go' plugin to build the software.
 plugin: go
 # Fetch the source code from Git at the 'v0.17.0' tag.
 source: https://github.com/kubernetes/release
 source-type: git
 source-tag: v0.17.8
 # Change to the specified sub-directory for the build.
 source-subdir: images/build/go-runner
 # Install the following snaps in the build environment.
 build-snaps:
 - go/1.20/stable
 # Set the following environment variables in the build
 # environment.
 build-environment:
 - CGO_ENABLED: 0
 - GOOS: linux

This instructs rockcraft to fetch a Git repository at a particular tag, change into the sub-directory images/build/go-runner, then build the software using the go plugin. It also specifies that the build required the go snap from the 1.20/stable track, and sets some environment variables. That’s a lot of result for not much YAML. The end result of this is a single binary that’s “staged” and ready to be placed (in this case) into a Rock (Canonical’s name for OCI images).

And the best part: this exact definition can be used in a rockcraft.yaml when building a Rock, a snapcraft.yaml when building a Snap, a charmcraft.yaml when building a Charm, etc.

The plugin system is extensive: at the time of writing there are 22 supported plugins, including go, maven, uv, meson and more. If your build system of choice isn’t supported you can specify manual steps, giving you as much flexibility as you need:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


wasi-sdk:
 # There is no appropriate plugin for this part, so set
 # it to 'nil' and we'll specify our own build process
 # using 'override-build'.
 plugin: nil
 # In this recipe, a previous part named 'clang' is
 # required to build before attempting to build this
 # part.
 after:
 - clang
 # Specify any `apt` packages required in the build
 # environment.
 build-packages:
 - wget
 # Set some environment variables for the build
 # environment.
 build-environment:
 - WASI_BRANCH: "15"
 - WASI_RELEASE: "15.0"
 # Define how to pull the software manually.
 override-pull: |
 ROOT=https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-$WASI_BRANCH
 wget $ROOT/wasi-sysroot-$WASI_RELEASE.tar.gz
 wget $ROOT/libclang_rt.builtins-wasm32-wasi-$WASI_RELEASE.tar.gz
 # Define how to 'build' the software manually
 override-build: |
 craftctl default
 tar -C $CRAFT_STAGE -xf wasi-sysroot-$WASI_RELEASE.tar.gz
 tar -C $CRAFT_STAGE/usr/lib/clang/* -xf libclang_rt.builtins-wasm32-wasi-$WASI_RELEASE.tar.gz
 # Don't prime anything for inclusion in the
 # final artifact; this part is only used for
 # another part's build process.
 override-prime: ''

Here, multiple stages of the lifecycle are overridden using override-build, override-pull and override-stage, and we see craftctl default for the first time, which instructs snapcraft to do whatever it would have done prior being overridden, but allows the developer to provide additional steps either before or after the default actions.

Isolated build environments #

Even once a recipe for building software is defined, preparing machines to build software can be painful. Different major versions of the same OS might have varying package availability, your team might run completely different operating systems, and you might have limited image availability in your CI environment.

The crafts solve this with build “backends”. Currently the crafts can use LXD or Multipass to create isolated build environments, which makes it work nicely on Linux, macOS and Windows. This functionality is handled automatically by the crafts through the craft-providers library. The craft-providers library provides uniform interfaces for creating build environments, configuring base images and executing builds.

This means if you can run snapcraft pack on your machine, your teammates can also run the same command without worrying about installing the right dependencies or polluting their machines with software and temporary files that might result from the build.

One of my favourite features of this setup is the ability to drop into a shell inside the build environment automatically on a few different conditions:

1
2
3
4
5
6


# Drop into a shell if any part of the build fails.
snapcraft pack --debug
# Drop into a shell after the build stage.
rockcraft build --shell-after
# Drop to a shell in lieu of the prime stage.
snapcraft prime --shell

This makes troubleshooting a failing build much simpler, while allowing the developer to maintain a clean separation between the build environment and their local machine. Should the build environment ever become polluted, or otherwise difficult to work with, you can always start from a clean slate with snapcraft|rockcraft|charmcraft clean. Each build machine is constructed using a cached build-base, which contains all the baseline packages required by the craft - so recreating the build environment for a specific package only requires that base to be cloned and augmented with project specific concerns - making the process faster.

Saving space #

When packaging any kind of software, a common concern is the size of the artifact. This might be because you’re building an OCI-image that is pulled thousands of times a day as part of a major SaaS deployment, or maybe it’s a Snap for an embedded device running Ubuntu Core with a limited flash. In the container world, “distroless” became a popular way to solve this problem - essentially popularising the practice of shipping the barest minimum in a container image, eschewing much of the traditional Unix FHS.

The parts mechanism has provided a way of “filtering” what is staged or primed into a final artifact from the start, which already gave developers autonomy to choose exactly what went into their builds.

In addition to this, Canonical built “chisel”, which extends the distroless concept beyond containers to any kind of artifact. With chisel, developers can slice out just the binaries, libraries, and configuration files they need from the Ubuntu Archive, enabling ultra-small packages without losing the robustness of Ubuntu’s ecosystem.

We later launched Chiseled JRE containers, and there are numerous other Rocks that utilise chisel to provide a balance between shipping tiny container images, while benefiting from the huge selection and quality of software in the Ubuntu Archive.

Because the crafts are all built on a common platform, they now all have the ability to use “slices” from chisel-releases, which enables a greater range of use-cases where artifact size is a primary concern. Slices are community maintained, and specified in simple to understand YAML files. You can see the list of available slices for the most recent Ubuntu release (25.04 Plucky Puffin) on GitHub, and further documentation on slices and how they’re used in the Chisel docs.

Multi-architecture builds #

Ubuntu supports six major architectures at the time of writing (amd64, arm64, armhf, ppc64le, s390x, riscv64), and all of our crafts have first-class support for each of them. This functionality is provided primarily by the craft-platforms library, and supported by the craft-grammar library, which enables more complex definitions where builds may have different steps or requirements for different architectures.

At a high-level, each artifact defines which architectures or platforms it is built for, and which it is built on. These are often, but not always, the same. For example:

1
2


platforms:
 amd64:

This is shorthand for “build the project on amd64 for amd64”, but in a different example taken from a charmcraft.yaml

1
2
3
4


platforms:
 all:
 build-on: [amd64]
 build-for: [all]

In this case the software is built on amd64, but can run on any of the supported architectures - this can happen with all-Python wheels, bash scripts and other interpreted languages which don’t link platform-specific libraries.

In some build processes, the process or dependencies might differ per-architecture, which is where craft-grammar comes in, enabling expressions such as (taken from GitHub):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


fit-image:
 # ...
 build-packages:
 # ...
 - wget
 - libjson-c-dev:${CRAFT_ARCH_BUILD_FOR}
 - libcryptsetup-dev:${CRAFT_ARCH_BUILD_FOR}
 # Only use the following build packages when building for armhf
 - to armhf:
 - binutils-arm-linux-gnueabi
 - gcc-arm-linux-gnueabihf
 - pkgconf:armhf
 # When building for arm64, use a different set
 - to arm64:
 # Dependencies for building *for* arm64 *on* amd64!
 - on amd64:
 - gcc-aarch64-linux-gnu
 - pkgconf:arm64
 - on arm64:
 - gcc

Being able to define how to build on different architectures is only half of the battle, though. It’s one thing to define how to build software on an s390x machine but few developers have mainframes handy to actually run the build! This is where the crafts’ remote-build capability comes in. The remote-build command sends builds to Canonical’s build farm, which has native support for all of Ubuntu’s supported architectures. This is built into all of our crafts, and is triggered with snapcraft remote-build, rockcraft remote-build, etc.

Remote builds are a lifeline for publishers and communities who need to reach a larger audience, but can’t necessarily get their own build farm together. One example of this is Snapcrafters, a community-driven organisation that packages popular software as Snaps, who use remote-build to drive multi-architecture builds from GitHub Actions as part of their publishing workflow (as seen here and here for example).

Unified testing framework #

Testing is often the missing piece in build tools: developers are forced to rely on separate CI systems or ad-hoc scripts to verify their artifacts. To close this gap, we’re introducing a unified test sub-command in the crafts.

We recently added the test sub-command to our crafts as an experimental (for now!) feature. Under the hood, craft test will introduce a new lifecycle stage (TEST). The enables packagers of any artifact type to specify how that artifact should be tested using a common framework across artifact types.

Craft’s testing capability is powered by spread, a convenient full-system task distribution system. Spread was built to simplify the massive number of integration tests run for the snapd project. It enables developers to specify tests in a simple language, and distribute them concurrently to any infrastructure they have available.

This enables a developer to define tests and test infrastructure, and make it trivial to run the same tests locally, or remotely on cloud infrastructure. This can really speed up the development process - preventing developers from needing to wait on CI runners to spin up and test their code while iterating, they can run the very same integration tests locally using craft test.

There are lots of fine details to spread, and the team is working on artifact-specific abstractions for the crafts that will make testing delightful. Imagine maintaining the Snap for a GUI application, and being able to enact the following workflow:

1
2
3
4
5
6
7
8
9


# Pull the repository
git clone https://github.com/some-gui-app/snap && cd snap
# Make some changes, perhaps fix a bug
vim snap/snapcraft.yaml
# Build the snap, and run the integration tests.
# These tests might include spinning up a headless
# graphical VM, which actually installs and runs
# the snap, and interacts with it
snapcraft test

By integrating a common testing tool into the build tooling, the Starcraft team will be able to curate unique testing experiences for each kind of artifact. A snap might need a headless graphical VM, where an OCI-image simply requires a container runtime, but the spread underpinnings allow a common test-definition language for each.

There are a couple of examples of this in the wild already:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# Install charmcraft
sudo snap install --classic charmcraft
# Clone the repo
git clone https://github.com/jnsgruk/zinc-k8s-operator
cd zinc-k8s-operator
# List the available tests
charmcraft test --list lxd:
# Run the integration testing suite, spinning up
# a small VM, inside which is a full Kubernetes
# instance, with a Juju controller bootstrapped.
# From here the charm will be deployed and tested to
# ensure it's integrations with the observability
# stack and ingress charms are functioning correctly.
charmcraft test -v lxd:ubuntu-24.04:tests/spread/observability-relations:juju_3_6

The test above is powered by this spread.yaml, and this test definition. With a little bit of work, it’s also possible to integrate spread with GitHub matrix actions, giving you one GitHub job per spread test - as seen here.

You can see a similar example in our PostgreSQL Snap test suite, and we’ll be adding more and more of this kind of test across our Rock, Snap, Charm, Image and Deb portfolio.

There is work to do, but I’m really excited about bringing a common testing framework to the crafts which should make the testing of all kinds of artifacts more consistent and easier to integrate across teams and systems.

Crafting the crafts #

As the portfolio expanded from snapcraft, to charmcraft, to rockcraft and is now expanding further to debcraft and imagecraft it was clear that we’d need a way to make it easy to build crafts for different artifacts, while being rigorous about consistency across the tools. A couple of years ago, the team built the craft-application base library, which now forms the foundation of all our crafts.

The craft-application library combines many of the existing libraries that were in use across the crafts (listed below), providing a consistent base upon which artifact-specific logic can be built. The allows craft developers to spend less time implementing CLI details, parts lifecycles and store interactions, and more time on curating a great experience for the maintainers of their artifact type.

For the curious, craft-application builds upon the following libraries:

craft-archives: manages interactions with apt package repositories
craft-cli: CLI client builder that follows the Canonical’s CLI guidelines
craft-parts: obtain, process, and organize data sources into deployment-ready filesystems.
craft-grammar: advanced description grammar for parts
craft-providers: interface for instantiating and executing builds for a variety of target environments
craft-platforms: manage target platforms and architectures for craft applications
craft-store: manage interactions with Canonical’s software stores
craft-artifacts: pack artifacts for craft applications

Examples and docs #

Before I leave you, I wanted to reference a few *craft.yaml examples, and link to the documentation for each of the crafts, where you’ll find the canonical (little c!) truth on each tool.

You can find documentation for the crafts below:

And some example recipes:

Snap: icloudpd - snapcraft.yaml
Snap: parca-agent - snapcraft.yaml
Snap: signal-desktop - snapcraft.yaml
Charm: ubuntu-manpages-operator - charmcraft.yaml
Rock: grafana - rockcraft.yaml
Rock: temporal-server - rockcraft.yaml

Summary #

The craft ecosystem provides developers with a rigorous, consistent and pleasant experience for building many kinds of artifacts. At the moment, we support Snaps, Rocks and Charms but we’re actively developing crafts for Debian packages, cloud images and more.The basic build process, parts ecosystem and foundations of the crafts are “battle tested” at this point, and I’m excited to see how the experimental craft test commands shape up across the crafts.

One of the killer features for the crafts is the ability to reuse part definitions across different artifacts - which makes the pay off for learning the parts language very high - it’s a skill you’ll be able to use to build Snaps, Rocks, Charms, VM Images and soon Debs!

If I look at ecosystems like Debian, where tooling like autopkgtest is the standard, I think debcraft test will offer an intuitive entrypoint and encourage more testing, and the same is true of Snaps, both graphical and command-line.

That’s all for now!

Simplifying Test & Release of Snapped GUI Apps

Mon, 18 Mar 2024 00:00:00 +0000

Introduction #

For the past few months, I’ve been getting steadily more involved in Snapcrafters. The Snapcrafters are a community dedicated to the creation and maintenance of ~100 snap packages. They’re currently maintaining snaps for applications like Signal Desktop, Discord, Gimp, Terraform and many more, some with hundreds of thousands of weekly active users. As with any community organisation, maintainer participation can ebb and flow over time as people find themselves with competing priorities.

One of my personal goals for participation in the Snapcrafters org was to help them build more automated, sustainable processes for bumping versions of snaps as the upstreams move forward, and find a more robust way to test GUI applications before they’re released to the masses.

The snap store comes with a surprisingly rich delivery mechanism consisting of tracks, risks and branches. This means that (among other things) changes to applications can be tested by way of an incremental roll out by those willing to help out - by subscribing to the edge or candidate channels.

The Problem #

While bumping the versions of the applications in snap packages can be done trivially, testing that the new application can launch on the Linux desktop and function correctly is more difficult - especially given the “headless” nature of CI systems, and the inherent complexity of some of the applications involved.

Electron has, in my opinion, been a huge net win for the Linux desktop. Performance and early Wayland compatibility aside, the selection of mainstream applications available to the Linux desktop user is certainly much greater as a result. One downside is that each app is essentially a browser with lots of complex moving parts which can be difficult to maintain for packagers over time - and particularly so for a team of volunteers who may not be experts in the applications they help to maintain.

In a previous post I wrote about how KVM-acceleration is now available on Github Actions runners. While it’s certainly possible to just pull in various pieces of the Linux desktop using apt directly on a Github Actions runner, the resulting configuration normally involves convoluted setup with VNC or similar. Such setups are usually fragile, and can be more difficult to reproduce locally - making it slower to debug any issues that do arise.

LXD Desktop VMs #

Before working at Canonical, I must confess to having paid very little attention to LXD. Since joining, it’s become one of my most used tools in my daily workflow. I had some early experience with LXD a few years ago when it essentially “just” did Ubuntu containers, but it’s evolved into a very competent hypervisor in its own right, providing both container and virtual machine images for numerous different Linux distributions. In more recent history, desktop virtual machines were introduced which gives a very fast way to boot into a desktop across multiple distributions.

To boot into a Ubuntu 22.04 LTS desktop virtual machine, for example:

1

lxc launch images:ubuntu/22.04/desktop ubuntu --vm --console=vga

As a side note, last year Canonical announced the LXD UI, which is a beautiful web UI for managing clusters of LXD servers, and includes a graphical web console for virtual machines - which is incredibly useful for testing software across versions and desktops.

Wrapping LXD #

After playing with LXD a bunch locally, I confirmed that between lxc exec, lxc file pull I had all I need. The commands were all relatively simple, but I wanted to ensure that the Github Actions I wrote were as maintainable as possible, so I decided to write a small wrapper for LXD, which ended up being named ghvmctl (Github Virtual Machine Control) because naming is…. hard!

There is nothing special about ghvmctl, it is just a bash script. Perhaps one day I’ll implement it in something a little more… rigorous? That said, it’s wrapping relatively few shell commands, with very few variables, and it’s been solid for several months now. The sum of ghvmctl’s capabilities can be summarised as:

Launch Ubuntu desktop VMs
Dismiss any initial setup wizards
Ensure that gnome-screenshot is installed
Provide a simple way to install and run snaps
Provide a simple way to screenshot the whole screen, and the active window

The script is mostly contained in a single file, apart from ghvmctl-runner which is pushed automatically into any VMs started by ghvmctl, and provides a way for applications to be run with all the appropriate environment variables such that graphical applications can run when the VM is being controlled headlessly (such as DISPLAY, WAYLAND_DISPLAY, XDG_SESSION_TYPE, etc.).

There are very few dependencies for the script, but I decided to package it as a snap to simplify installing it on Github runners. The snap is simple, containing just the two scripts mentioned above, and the LXC client. This also means you can install and use the tool locally should you wish to experiment with it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


# Install ghvmctl
sudo snap install ghvmctl
# Allow ghvmctl to access the LXD socket
sudo snap connect ghvmctl:lxd lxd:lxd
# Launch a VM and prepare for testing
ghvmctl prepare
# Install a snap from the store
ghvmctl snap-install signal-desktop
# Run the snap on the desktop
ghvmctl snap-run signal-desktop

# Wait a few seconds for the app to start...

# Take screenshots and pull them back to $HOME/ghvmctl-screenshots
ghvmctl screenshot-full
ghvmctl screenshot-window

To simplify the installation and setup of ghvmctl, there is also a Github Action which takes care of enabling KVM on the runner, initialising LXD, installing ghvmctl and ensuring it has access to the LXD socket.

Building An Integrated Workflow #

By now, the Snapcrafters have quite a sophisticated collection of Github Actions which are used for managing the release lifecycle of snaps, but for me it was this piece that tied it all together for GUI applications. The actions can be summarised as:

Parsing snapcraft.yaml files for details such as name, architectures, version
Building snaps locally on Github Actions runners when Pull Requests are made
Building snaps across architectures on the Launchpad build farm when changes are merged
Create a “Call for Testing” Github Issue with details of the candidate revisions
Follow up on the issue with screenshots in a comment
Promote the snap to stable when a maintainer issues the command

None of these things are particularly hard in their own right, but they amount to some complicated juggling of Github Actions artefacts and tokens for various repositories and external services.

Building A Screenshot Action #

The Github Action responsible for collecting screenshots is used across multiple snaps to give maintainers a bit more confidence when releasing changes into the stable channels for their snaps.

The action makes use of ghvmctl to launch VMs, install the candidate snaps and collect screenshots of them. This turned out to be simple with the introduction of ghvmctl - the complicated part turned out to be where to store the screenshots such that they could be published in a comment! Github provides image/file hosting for comments on issue when the comments are made through the web UI. As far as I can tell, there is no way to submit a comment with an embedded picture from the Github API (let me know!), and I wasn’t keen to rely on a third party service such as Imgur.

The solution we settled on was to create a ci-screenshots, which could be published to by the workflows of each snap repository. We will, over time, clear out older screenshots.

End Result #

An example of the end result can be seen in this Github Issue. Let’s break down what happened.

First, once a change was merged into the Signal Desktop snap repository, and the resulting snap was built for each of its target architectures:

As part of this process an issue was automatically created containing information about the new candidate versions:

A couple of minutes later, the bot followed up with a comment containing screenshots of the application running on the desktop:

Once I was content that the snap was working correctly, and I’d tested the revision out locally, I then issued a command to promote the snap into the stable channel, where it was slowly rolled out across the user base:

You can see examples of this working across multiple snaps:

Summary #

We’re still in the process of refining and rolling out this process, but I hope that it will help reduce burden on maintainers over time, and result in fresher and more reliable desktop snaps in the Snap Store. If you’d like to get involved in Snapcrafters, reach out to me, post on the Snapcraft Discourse or join the Matrix room.