Testing on Jon Seager

Simplifying Test & Release of Snapped GUI Apps

Mon, 18 Mar 2024 00:00:00 +0000

Introduction #

For the past few months, I’ve been getting steadily more involved in Snapcrafters. The Snapcrafters are a community dedicated to the creation and maintenance of ~100 snap packages. They’re currently maintaining snaps for applications like Signal Desktop, Discord, Gimp, Terraform and many more, some with hundreds of thousands of weekly active users. As with any community organisation, maintainer participation can ebb and flow over time as people find themselves with competing priorities.

One of my personal goals for participation in the Snapcrafters org was to help them build more automated, sustainable processes for bumping versions of snaps as the upstreams move forward, and find a more robust way to test GUI applications before they’re released to the masses.

The snap store comes with a surprisingly rich delivery mechanism consisting of tracks, risks and branches. This means that (among other things) changes to applications can be tested by way of an incremental roll out by those willing to help out - by subscribing to the edge or candidate channels.

The Problem #

While bumping the versions of the applications in snap packages can be done trivially, testing that the new application can launch on the Linux desktop and function correctly is more difficult - especially given the “headless” nature of CI systems, and the inherent complexity of some of the applications involved.

Electron has, in my opinion, been a huge net win for the Linux desktop. Performance and early Wayland compatibility aside, the selection of mainstream applications available to the Linux desktop user is certainly much greater as a result. One downside is that each app is essentially a browser with lots of complex moving parts which can be difficult to maintain for packagers over time - and particularly so for a team of volunteers who may not be experts in the applications they help to maintain.

In a previous post I wrote about how KVM-acceleration is now available on Github Actions runners. While it’s certainly possible to just pull in various pieces of the Linux desktop using apt directly on a Github Actions runner, the resulting configuration normally involves convoluted setup with VNC or similar. Such setups are usually fragile, and can be more difficult to reproduce locally - making it slower to debug any issues that do arise.

LXD Desktop VMs #

Before working at Canonical, I must confess to having paid very little attention to LXD. Since joining, it’s become one of my most used tools in my daily workflow. I had some early experience with LXD a few years ago when it essentially “just” did Ubuntu containers, but it’s evolved into a very competent hypervisor in its own right, providing both container and virtual machine images for numerous different Linux distributions. In more recent history, desktop virtual machines were introduced which gives a very fast way to boot into a desktop across multiple distributions.

To boot into a Ubuntu 22.04 LTS desktop virtual machine, for example:

1

lxc launch images:ubuntu/22.04/desktop ubuntu --vm --console=vga

As a side note, last year Canonical announced the LXD UI, which is a beautiful web UI for managing clusters of LXD servers, and includes a graphical web console for virtual machines - which is incredibly useful for testing software across versions and desktops.

Wrapping LXD #

After playing with LXD a bunch locally, I confirmed that between lxc exec, lxc file pull I had all I need. The commands were all relatively simple, but I wanted to ensure that the Github Actions I wrote were as maintainable as possible, so I decided to write a small wrapper for LXD, which ended up being named ghvmctl (Github Virtual Machine Control) because naming is…. hard!

There is nothing special about ghvmctl, it is just a bash script. Perhaps one day I’ll implement it in something a little more… rigorous? That said, it’s wrapping relatively few shell commands, with very few variables, and it’s been solid for several months now. The sum of ghvmctl’s capabilities can be summarised as:

Launch Ubuntu desktop VMs
Dismiss any initial setup wizards
Ensure that gnome-screenshot is installed
Provide a simple way to install and run snaps
Provide a simple way to screenshot the whole screen, and the active window

The script is mostly contained in a single file, apart from ghvmctl-runner which is pushed automatically into any VMs started by ghvmctl, and provides a way for applications to be run with all the appropriate environment variables such that graphical applications can run when the VM is being controlled headlessly (such as DISPLAY, WAYLAND_DISPLAY, XDG_SESSION_TYPE, etc.).

There are very few dependencies for the script, but I decided to package it as a snap to simplify installing it on Github runners. The snap is simple, containing just the two scripts mentioned above, and the LXC client. This also means you can install and use the tool locally should you wish to experiment with it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


# Install ghvmctl
sudo snap install ghvmctl
# Allow ghvmctl to access the LXD socket
sudo snap connect ghvmctl:lxd lxd:lxd
# Launch a VM and prepare for testing
ghvmctl prepare
# Install a snap from the store
ghvmctl snap-install signal-desktop
# Run the snap on the desktop
ghvmctl snap-run signal-desktop

# Wait a few seconds for the app to start...

# Take screenshots and pull them back to $HOME/ghvmctl-screenshots
ghvmctl screenshot-full
ghvmctl screenshot-window

To simplify the installation and setup of ghvmctl, there is also a Github Action which takes care of enabling KVM on the runner, initialising LXD, installing ghvmctl and ensuring it has access to the LXD socket.

Building An Integrated Workflow #

By now, the Snapcrafters have quite a sophisticated collection of Github Actions which are used for managing the release lifecycle of snaps, but for me it was this piece that tied it all together for GUI applications. The actions can be summarised as:

Parsing snapcraft.yaml files for details such as name, architectures, version
Building snaps locally on Github Actions runners when Pull Requests are made
Building snaps across architectures on the Launchpad build farm when changes are merged
Create a “Call for Testing” Github Issue with details of the candidate revisions
Follow up on the issue with screenshots in a comment
Promote the snap to stable when a maintainer issues the command

None of these things are particularly hard in their own right, but they amount to some complicated juggling of Github Actions artefacts and tokens for various repositories and external services.

Building A Screenshot Action #

The Github Action responsible for collecting screenshots is used across multiple snaps to give maintainers a bit more confidence when releasing changes into the stable channels for their snaps.

The action makes use of ghvmctl to launch VMs, install the candidate snaps and collect screenshots of them. This turned out to be simple with the introduction of ghvmctl - the complicated part turned out to be where to store the screenshots such that they could be published in a comment! Github provides image/file hosting for comments on issue when the comments are made through the web UI. As far as I can tell, there is no way to submit a comment with an embedded picture from the Github API (let me know!), and I wasn’t keen to rely on a third party service such as Imgur.

The solution we settled on was to create a ci-screenshots, which could be published to by the workflows of each snap repository. We will, over time, clear out older screenshots.

End Result #

An example of the end result can be seen in this Github Issue. Let’s break down what happened.

First, once a change was merged into the Signal Desktop snap repository, and the resulting snap was built for each of its target architectures:

As part of this process an issue was automatically created containing information about the new candidate versions:

A couple of minutes later, the bot followed up with a comment containing screenshots of the application running on the desktop:

Once I was content that the snap was working correctly, and I’d tested the revision out locally, I then issued a command to promote the snap into the stable channel, where it was slowly rolled out across the user base:

You can see examples of this working across multiple snaps:

Summary #

We’re still in the process of refining and rolling out this process, but I hope that it will help reduce burden on maintainers over time, and result in fresher and more reliable desktop snaps in the Snap Store. If you’d like to get involved in Snapcrafters, reach out to me, post on the Snapcraft Discourse or join the Matrix room.

Integration testing with NixOS in Github Actions

Sat, 10 Feb 2024 00:00:00 +0000

Introduction #

While in my own time I’ve tended toward NixOS over the past 18 months, in my day-to-day work for Canonical I’m required to interact with a fair few of our products - and particularly build tools.

I frequently need to use some combination of Snaps, Charms and Rocks. Each of these have their own “craft” build tools (snapcraft, charmcraft and rockcraft), which are distributed exclusively as Snap packages and thus a little tricky to consume from NixOS.

The Problem #

Packaging the tools for Nix was a little repetitive, but not particularly difficult. They’re all built with Python, and share a common set of libraries. Testing that the packages were working correctly (i.e. could actually build software) on NixOS using the NixOS version of LXD in Github Actions proved more difficult.

Github Actions defaults to Ubuntu as the operating system for its runners - an entirely sensible choice, but not one that was going to help me test packages could work together on NixOS.

I could have hosted my own Github Actions runners to solve the problem, but I didn’t want to maintain such a deployment.

For a while I relied on just testing each of the crafts locally before pushing, and the CI simply installed the Nix package manager on the runners (using the excellent Nix installer from Determinate Systems) and ensured that the build could succeed, but this left a lot to be desired - particularly when I accidentally (and somewhat inevitably) broke one of the packages.

KVM for Github Actions #

Some time later I came across this post on the Github Blog, stating the following:

Starting on February 23, 2023, Actions users […] will be able to make use of hardware acceleration […].

What follows is an example of a relatively simple addition to a Github Workflow to enable KVM on Github Actions runners:

1
2
3
4
5


- name: Enable KVM group perms
 run: |
 echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules
 sudo udevadm control --reload-rules
 sudo udevadm trigger --name-match=kvm

Given the ability to relatively easily create NixOS VMs from a machine configuration, this should enable me to run a NixOS VM inside my Github Actions runners, and use that VM to run end to end tests of my craft packages.

After some quick tests, I confirmed that the above snippet worked just fine on the freely available runners that are assigned to public projects. After tooting excitedly about this, it was also picked up by the folks at Determinate Systems who promptly added support for this in their Nix install Github Action - enabling the feature by default.

Building VMs with Nix #

A really nice feature of NixOS that I discovered relatively late, is that given a NixOS machine configuration it’s trivial to build a virtual machine image for that configuration. This has the nice property that one can actually boot a VM-equivalent of any previously defined machines. You could, for example, boot a VM-equivalent of my laptop with the following command:

1

nix run github:jnsgruk/nixos-config#nixosConfigurations.freyja.config.system.build.vm

In order to test my craft tools, I needed a relatively simple NixOS VM that had LXD enabled, and my craft tools installed. My test VM configuration looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45


{ modulesPath, flake, pkgs, ... }: {
 # A nice helper that handles creating the VM launch script, which in turn
 # ensures the disk image is created as required, and QEMU is launched
 # with sensible parameters.
 imports = [ "${modulesPath}/virtualisation/qemu-vm.nix" ];

 # Define the version of NixOS and the architecture.
 system.stateVersion = "23.11";
 nixpkgs.hostPlatform = "x86_64-linux";

 # This overlay is provided by the crafts-flake, and ensures that
 # 'pkgs.snapcraft', 'pkgs.charmcraft', 'pkgs.rockcraft' all resolve to
 # the packages in the flake.
 nixpkgs.overlays = [ flake.outputs.overlay ];

 # These values are tuned such that the VM performs on Github Actions runners.
 virtualisation = {
 forwardPorts = [{ from = "host"; host.port = 2222; guest.port = 22; }];
 cores = 2;
 memorySize = 5120;
 diskSize = 10240;
 };

 # Configure the root user without password and enable SSH.
 # This VM will only ever be used in short-lived testing environments with
 # no inbound networking permitted, so there is minimal (if any) risk.
 # If you put this VM on the internet, you can keep the pieces! :)
 networking.firewall.enable = false;
 services.openssh.enable = true;
 services.openssh.settings.PermitRootLogin = "yes";
 users.extraUsers.root.password = "password";

 # Ensure that LXD is installed, and started on boot.
 virtualisation.lxd.enable = true;

 # Include the `craft-test` script, ensuring the craft apps are installed
 # and included in its PATH.
 environment.systemPackages = [
 (pkgs.writeShellApplication {
 name = "craft-test";
 runtimeInputs = with pkgs; [ unixtools.xxd git snapcraft charmcraft rockcraft ];
 text = builtins.readFile ./craft-test;
 })
 ];
}

Anybody can build and launch this VM trivially:

1

nix run github:jnsgruk/crafts-flake#testVM

Writing a Github workflow #

All the building blocks are in place! I wanted to keep the actual workflow definition for the tests as clean and understandable as possible, so I put together the craft-test script as a small helper which automates the building of real artefacts. An example invocation might be:

1

bash craft-test snapcraft

On each invocation, the script creates temporary directory, clones some representative build files for the selected craft tool, and launches the craft. The repos it uses for the representative packages are hard-coded for each craft for now.

I wrote one more small helper script to simplify connecting to the VM with the required parameters. It’s a wrapper around ssh and sshpass that’s hard-coded with the credentials of the test VM (don’t @ me!), and executes commands over SSH in the test VM. Using this script, one can bash vm-exec -- craft-test snapcraft and the craft-test script will be executed over SSH in the VM.

With all that said and done, the resulting workflow is pleasingly simple:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


# ...
jobs:
 test:
 runs-on: ubuntu-latest
 strategy:
 matrix:
 package: ["charmcraft", "rockcraft", "snapcraft"]
 steps:
 - name: Checkout flake
 uses: actions/checkout@v4

 - name: Install nix
 uses: DeterminateSystems/nix-installer-action@v9

 - name: Build and run the test VM
 run: |
 nix run .#testVm -- -daemonize -display none

 - name: Test ${{ matrix.package }}
 run: |
 nix run .#testVmExec -- craft-test ${{ matrix.package }}

A separate job is run for each of the crafts, and a real artefact is built in each, giving reasonable confidence that the consumers of my flake will be successful when building snaps, rocks and charms natively on NixOS. A successful run can be seen here.

Summary #

In this article we’ve covered:

Enabling KVM on Github Runners
Building NixOS VMs using Flakes
Booting NixOS VMs in Github Actions

If you’d like to build snaps, rocks or charms and you’re running NixOS, you can run the tools individually from my flake:

1
2
3
4
5
6
7
8


# Run charmcraft
nix run github:jnsgruk/crafts-flake#charmcraft

# Run rockcraft
nix run github:jnsgruk/crafts-flake#rockcraft

# Run snapcraft
nix run github:jnsgruk/crafts-flake#snapcraft

Or you can check out the README for instructions on how to integrate into your Nix config using overlays!

That’s all for now! 🤓