Github Actions on Jon Seager

Tracking Releases & CI Across Software Teams and Forges

Wed, 22 May 2024 00:00:00 +0000

Introduction #

In my day job at Canonical, I lead the teams developing Juju, and a whole host of charms. Charms are software packages used for deploying applications on any infrastructure you have available. The packages are portable, meaning you can use our PostgreSQL operator on AWS, on Azure, on Openstack, on Google Cloud, etc. We’re building up quite the portfolio of popular open source applications across data engineering, observability, identity, telco, MLOps and more. I won’t go into detail about Juju or charms in this post, but I likely will in a future post.

The important thing for this post is that I look after >10 software teams, who use two different software forges (Github and Launchpad), and all push artifacts into both the Snap Store and the Charmhub. I wanted a way to keep track of their releases, and provide a tool that my managers could use to do the same. Put simply, I wanted a unified view of:

The latest Github releases
The latest Launchpad tags
Channels in the Snap Store
Channels in Charmhub
CI status

In particular, I wanted an easy way to be able to tie releases/commits in a forge to specific revisions in our various stores. This would enable us to troubleshoot more easily when issues are reported in customer environments by making it easier to jump to the code for the specific revision they’re running.

At the time I started working on this, the effort at Canonical to build the portfolio of charms was really ramping up - and the only way of tracking which team owned which charm (between product teams, our IS department and our datacentre team) was a giant spreadsheet which was perpetually out of date (obviously!). The one thing that could be relied upon was which team owned a repo on Github or Launchpad - so my hope was to also reliably answer the question “which team owns the foo charm?”.

You can see the result of this effort at https://releases.juju.is, and a sneak peek below:

Prior Art #

I wasn’t the first to have these problems. The idea to start tracking information in this way came from the elementaryOS releases tracker. This was a tool they built for keeping track of their various repositories, and in particular those which had seen many commits since the last release. The code for the site is available on Github.

The elementaryOS tracker uses a Python script to scrape the Github API, which outputs a JSON representation of the state of their repositories. This is then parsed during the build of a Jekyll site, which is published on Github Pages using a Github Workflow.

This actually got me very close, and in fact my first attempt was a trivial fork of this project with a few slight modifications to the Python script. However, their tool wasn’t designed to be used across multiple distinct teams, and there are some limitations such as not rendering Markdown in the release notes.

I also wanted to make some stylistic changes. I’m not particularly familiar with Jekyll and have generally used Hugo for such tasks. I did maintain a Jekyll version for some months, but my lack of (recent) familiarity with the Ruby & gems ecosystem was making updates and maintenance more tedious than I liked.

Another tool I came across when I was looking to solve this was willow by Amolith. According to the README:

Willow helps developers, sysadmins, and homelabbers keep up with software releases across arbitrary forge platforms, including full-featured forges like GitHub, GitLab, or Forgejo as well as more minimal options like cgit or stagit.

This is super close - but it doesn’t have the snap/charm tracking I sought, and probably won’t do given the scope of the project.

`releasegen` #

As I started to modify the Python script used in the elementaryOS release tracker, it started to get quite large, and I became a little uncomfortable with the state of it. At the time I was reintroducing myself into Go after a couple of years out, and decided that I would essentially “start again” and write a tool for my release tracker from scratch.

What I came up with is releasegen. A terribly boring and unimaginative name that I originally intended to change, but never got around to! Nevertheless, releasegen solved a number of problems for me even in its first release. The first important change was support for separating releases across multiple Github organisations and teams, through the use of a simple config format:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


teams:
 - name: Frontend
 github:
 - org: acme-corp
 teams:
 - frontend-bots
 ignores:
 - some-old-project

 - name: Backend
 github:
 - org: acme-corp
 teams:
 - backend-engineers

 - name: Packaging
 launchpad:
 project-groups:
 - acme-corp-debs

You can see the config file used to generated releases.juju.is on Github.

The output format is heavily inspired by that of the original Python script, but has been enriched with a few more fields over time.

This tool is really quite simple: it takes a config file which points it at a combination of Github Orgs/Teams and Launchpad project groups, and outputs a big JSON file containing details of releases and associated packages. You can see an example of the output in the charm-eng-releases repo.

One of the main reasons I chose Go in the first place was because of it’s great support for concurrency. Data from Github is gathered using the google/go-github package, which provides an interface to the Github API. When I first started adding repos to the original Python version, the run time quickly grew to 10+ minutes. My intention was to spin up a goroutine per repository in releasegen, but I quickly ran into secondary rate-limits. It turns out that the Github API prohibits you from making too many concurrent requests.

For Launchpad, things are more complicated. There is an API but it lacks methods for grabbing tags, specific commits etc. I also wanted to avoid cloning all of the repos for which information is gathered. The Launchpad code browsing UI is based on cgit, and hasn’t fundamentally changed in a long time. For now, releasegen relies upon scraping the Launchpad web pages (using goquery) to get the information it needs. This is not ideal, but has been functioning better than you might expect for around 18 months. There’s also no limit on the requests that can be made to the web frontend of Launchpad - so processing is able to be done concurrently across repos in this case.

In more recent times, releasegen grew the ability to read badges out of project READMEs, and use those badges to link repos to a particular store (either the Snap store or the Charmhub). Support for reading badges and parsing Github CI badges was kindly contributed by one of my colleagues. I have subsequently generalised that initial implementation and added support for the Snap store too.

Parsing `releasegen` with Hugo #

Pleasingly, Hugo has the ability to read data from YAML, JSON, XML, or TOML files, which can then be rendered throughout the site.

Given that this tool was to be used for tracking releases from Canonical teams, I created a basic layout using the excellent Vanilla Framework. A layout in Hugo is nothing more than a collection of HTML/CSS/JS into which data can be rendered. The entry point for that in my project is this index.html. Here I layout the basic structure of the page, and use a number of partials to render common components across the site. There are very few adjustments to the standard Vanilla Framework style, and a couple of small Javascript files to provide tabs, modals, expanding table rows, table sorting and suchlike.

Aside from the big screenshot at the start of this post, I want to highlight a couple of details in the UI. Firstly, if there is an associated artifact for a given repository (a Snap or a Charm), the row can be expanded to show details of that artifact:

Each individual release can also be expanded using the “eye” icon, at which point a modal will be displayed containing the release notes, complete with links:

There is also some visual distinction between those repos on Github, and those on Launchpad:

Publishing with Github Actions #

The site is hosted on Github Pages, and if you’ve been following along, you’ll notice that the site would need to be “regenerated” for the information to remain fresh - Hugo is ultimately a static site generator. This problem is solved in my case with Github Actions. I created a workflow which is triggered every hour to dump the latest report using releasegen, regenerate the Hugo site, and commit the outcome to branch of the repo which is used to serve the page. The workflow itself is pretty simple, and can be seen on Github.

Summary #

This tool scratched an itch for me. It’s a little flawed in places: it doesn’t update in real time, releasegen could do with some ~~more~~ tests, and I’m not super proud of how it parses data about Launchpad projects. That said, it’s been fairly reliable over the past 18 months, and it does present information in a pretty consistent way (though I’m no designer!). I personally think it’s a good example of how things can be automated and simplified by combining existing tools in a short space of time.

I have an idea about how to generalise the processing of data, which would both remove my reliance on the Github API, and also unify the approach for gathering information about Git repos across forges (making it easier to support the likes of Codeberg and Sourcehut). I read an interesting blog recently about Serving a Website from a Git Repo Without Cloning It which implies that a lot of the information I require such as commits, tags, etc. could be gleaned directly from the git endpoint over HTTP, but I haven’t yet looked in detail.

It did solve the problem of keeping a big spreadsheet up to date as a means of tracking project ownership, and it’s certainly proved a useful tool for me to understand the relationship between commits, Github Releases and released revisions in our stores across teams. Some of my managers/seniors have found it really valuable, others not so much - this is generally reflected by how complete the information is for each team’s repos. Over the past few months, I’ve had a couple of teams reach out to me and ask to be added despite them not being part of my org, because they see it as a useful tool, so there’s that!

This tooling was also adopted by the Snapcrafters who run their own version of the dashboard. There are some subtle differences here - the UI in this case also displays each snap’s base (e.g. core18, core22, etc.). You can see the source code for that on Github.

That’s all for now! If you’ve built something similar or you think I’ve missed a trick, let me know!

Simplifying Test & Release of Snapped GUI Apps

Mon, 18 Mar 2024 00:00:00 +0000

Introduction #

For the past few months, I’ve been getting steadily more involved in Snapcrafters. The Snapcrafters are a community dedicated to the creation and maintenance of ~100 snap packages. They’re currently maintaining snaps for applications like Signal Desktop, Discord, Gimp, Terraform and many more, some with hundreds of thousands of weekly active users. As with any community organisation, maintainer participation can ebb and flow over time as people find themselves with competing priorities.

One of my personal goals for participation in the Snapcrafters org was to help them build more automated, sustainable processes for bumping versions of snaps as the upstreams move forward, and find a more robust way to test GUI applications before they’re released to the masses.

The snap store comes with a surprisingly rich delivery mechanism consisting of tracks, risks and branches. This means that (among other things) changes to applications can be tested by way of an incremental roll out by those willing to help out - by subscribing to the edge or candidate channels.

The Problem #

While bumping the versions of the applications in snap packages can be done trivially, testing that the new application can launch on the Linux desktop and function correctly is more difficult - especially given the “headless” nature of CI systems, and the inherent complexity of some of the applications involved.

Electron has, in my opinion, been a huge net win for the Linux desktop. Performance and early Wayland compatibility aside, the selection of mainstream applications available to the Linux desktop user is certainly much greater as a result. One downside is that each app is essentially a browser with lots of complex moving parts which can be difficult to maintain for packagers over time - and particularly so for a team of volunteers who may not be experts in the applications they help to maintain.

In a previous post I wrote about how KVM-acceleration is now available on Github Actions runners. While it’s certainly possible to just pull in various pieces of the Linux desktop using apt directly on a Github Actions runner, the resulting configuration normally involves convoluted setup with VNC or similar. Such setups are usually fragile, and can be more difficult to reproduce locally - making it slower to debug any issues that do arise.

LXD Desktop VMs #

Before working at Canonical, I must confess to having paid very little attention to LXD. Since joining, it’s become one of my most used tools in my daily workflow. I had some early experience with LXD a few years ago when it essentially “just” did Ubuntu containers, but it’s evolved into a very competent hypervisor in its own right, providing both container and virtual machine images for numerous different Linux distributions. In more recent history, desktop virtual machines were introduced which gives a very fast way to boot into a desktop across multiple distributions.

To boot into a Ubuntu 22.04 LTS desktop virtual machine, for example:

1

lxc launch images:ubuntu/22.04/desktop ubuntu --vm --console=vga

As a side note, last year Canonical announced the LXD UI, which is a beautiful web UI for managing clusters of LXD servers, and includes a graphical web console for virtual machines - which is incredibly useful for testing software across versions and desktops.

Wrapping LXD #

After playing with LXD a bunch locally, I confirmed that between lxc exec, lxc file pull I had all I need. The commands were all relatively simple, but I wanted to ensure that the Github Actions I wrote were as maintainable as possible, so I decided to write a small wrapper for LXD, which ended up being named ghvmctl (Github Virtual Machine Control) because naming is…. hard!

There is nothing special about ghvmctl, it is just a bash script. Perhaps one day I’ll implement it in something a little more… rigorous? That said, it’s wrapping relatively few shell commands, with very few variables, and it’s been solid for several months now. The sum of ghvmctl’s capabilities can be summarised as:

Launch Ubuntu desktop VMs
Dismiss any initial setup wizards
Ensure that gnome-screenshot is installed
Provide a simple way to install and run snaps
Provide a simple way to screenshot the whole screen, and the active window

The script is mostly contained in a single file, apart from ghvmctl-runner which is pushed automatically into any VMs started by ghvmctl, and provides a way for applications to be run with all the appropriate environment variables such that graphical applications can run when the VM is being controlled headlessly (such as DISPLAY, WAYLAND_DISPLAY, XDG_SESSION_TYPE, etc.).

There are very few dependencies for the script, but I decided to package it as a snap to simplify installing it on Github runners. The snap is simple, containing just the two scripts mentioned above, and the LXC client. This also means you can install and use the tool locally should you wish to experiment with it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


# Install ghvmctl
sudo snap install ghvmctl
# Allow ghvmctl to access the LXD socket
sudo snap connect ghvmctl:lxd lxd:lxd
# Launch a VM and prepare for testing
ghvmctl prepare
# Install a snap from the store
ghvmctl snap-install signal-desktop
# Run the snap on the desktop
ghvmctl snap-run signal-desktop

# Wait a few seconds for the app to start...

# Take screenshots and pull them back to $HOME/ghvmctl-screenshots
ghvmctl screenshot-full
ghvmctl screenshot-window

To simplify the installation and setup of ghvmctl, there is also a Github Action which takes care of enabling KVM on the runner, initialising LXD, installing ghvmctl and ensuring it has access to the LXD socket.

Building An Integrated Workflow #

By now, the Snapcrafters have quite a sophisticated collection of Github Actions which are used for managing the release lifecycle of snaps, but for me it was this piece that tied it all together for GUI applications. The actions can be summarised as:

Parsing snapcraft.yaml files for details such as name, architectures, version
Building snaps locally on Github Actions runners when Pull Requests are made
Building snaps across architectures on the Launchpad build farm when changes are merged
Create a “Call for Testing” Github Issue with details of the candidate revisions
Follow up on the issue with screenshots in a comment
Promote the snap to stable when a maintainer issues the command

None of these things are particularly hard in their own right, but they amount to some complicated juggling of Github Actions artefacts and tokens for various repositories and external services.

Building A Screenshot Action #

The Github Action responsible for collecting screenshots is used across multiple snaps to give maintainers a bit more confidence when releasing changes into the stable channels for their snaps.

The action makes use of ghvmctl to launch VMs, install the candidate snaps and collect screenshots of them. This turned out to be simple with the introduction of ghvmctl - the complicated part turned out to be where to store the screenshots such that they could be published in a comment! Github provides image/file hosting for comments on issue when the comments are made through the web UI. As far as I can tell, there is no way to submit a comment with an embedded picture from the Github API (let me know!), and I wasn’t keen to rely on a third party service such as Imgur.

The solution we settled on was to create a ci-screenshots, which could be published to by the workflows of each snap repository. We will, over time, clear out older screenshots.

End Result #

An example of the end result can be seen in this Github Issue. Let’s break down what happened.

First, once a change was merged into the Signal Desktop snap repository, and the resulting snap was built for each of its target architectures:

As part of this process an issue was automatically created containing information about the new candidate versions:

A couple of minutes later, the bot followed up with a comment containing screenshots of the application running on the desktop:

Once I was content that the snap was working correctly, and I’d tested the revision out locally, I then issued a command to promote the snap into the stable channel, where it was slowly rolled out across the user base:

You can see examples of this working across multiple snaps:

Summary #

We’re still in the process of refining and rolling out this process, but I hope that it will help reduce burden on maintainers over time, and result in fresher and more reliable desktop snaps in the Snap Store. If you’d like to get involved in Snapcrafters, reach out to me, post on the Snapcraft Discourse or join the Matrix room.

Packaging Scrutiny for NixOS

Fri, 16 Feb 2024 00:00:00 +0000

Update: Since writing this post, I’ve contributed Scrutiny upstream to nixpkgs/NixOS. You can see the write up of that effort here.

Introduction #

In a recent (well, recent-ish) episode of the Self Hosted Show, there was some talk of a hard drive monitoring tool called Scrutiny. Scrutiny is a hard drive monitoring tool that exposes S.M.A.R.T data in a nice, clean dashboard. It gathers that S.M.A.R.T data using the venerable smartd, which is a Linux daemon that monitors S.M.A.R.T data from a huge number of ATA, IDE, SCSI-3 drives. The code is available on Github.

The aim of running such monitoring is to detect and replace failing hard drives before they cause an outage, or any data loss. Depending on the firmware of a drive, there are potentially hundreds of S.M.A.R.T attributes that can be collected, so it can be hard to understand which to pay attention to.

As good as smartd is, it has some shortcomings such as not recording historical data for each attribute, and relying upon value thresholds that are set by the manufacturers of the drive. Scrutiny aims to provide historical tracking of attribute readings, and thresholds based on real-world data from Backblaze.

This all looked very compelling, but it hasn’t been packaged for NixOS as far as I can tell. This post is quite “code heavy” and is intended as a bit of a deep-dive/walkthrough for people who are new to packaging in Nix. I’m certainly no expert here, and if you spot something I’ve done wrong - I’d love to hear about it!

Below is a screenshot of Scrutiny’s dashboard:

Architecture #

In order to get Scrutiny up and running, there were a few components that all needed packaging separately.

Scrutiny itself is made up of the following:

A web application backend - written in Go
A web application frontend - written in NodeJs/Angular
A collector service - written in Go

The web application is the dashboard that you’ll see the pretty screenshots of. The collector service is designed to be run on an interval to collect information from smartd, and send it to the web application’s API.

Scrutiny relies upon InfluxDB to store data about smart attributes in a timeseries.

Thinking about how to structure things, I decided that I would build two separate Nix packages: one for the dashboard and UI, and another for the collector. It seems one could run the collector and dashboard on different machines, so this seemed like a logical split.

Packaging for NixOS #

Packaging the Dashboard #

I decided to start with the web application. Because of how the project is laid out, this means combining two separate derivations based built from the same source code: one for the UI and one for the backend. I began by creating a file that would carry common attributes for each of the derivations, such as the version, repository information, hash, etc.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


# common.nix
{ pkgs , ... }:
let
 name = "scrutiny";
 version = "0.7.2";
in
{
 inherit name version;

 # Fetch the source code from Github, referring to
 # the version required by Git tag.
 repo = pkgs.fetchFromGitHub {
 owner = "AnalogJ";
 repo = name;
 rev = "v${version}";
 sha256 = "sha256-UYKi+WTsasUaE6irzMAHr66k7wXyec8FXc8AWjEk0qs=";
 };

 # ...
}

I then started packaging the frontend:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36


{ pkgs, ... }: {
 # Import some common attributes such
 # as name, version, src repo
 common = import ./common.nix { inherit pkgs; };
 inherit (common) name version repo vendorHash;

 # Create a package for the Javascript frontend
 frontend = pkgs.buildNpmPackage rec {
 inherit version;
 pname = "${name}-webapp-frontend";
 src = "${repo}/webapp/frontend";

 # This hash is generated at build time, and uniquely identifies
 # the cache of NodeJS dependencies used for the build.
 npmDepsHash = "sha256-M8P41LPg7oJ/C9abDuNM5Mn+OO0zK56CKi2BwLxv8oQ=";

 # Override the build phase to match the
 # upstream build process.
 buildPhase = ''
 runHook preBuild
 mkdir dist
 npm run build:prod --offline -- --output-path=dist
 runHook postBuild
 '';

 # Copy the output of the compiled javascript bundle
 # and site assets to the package output.
 installPhase = ''
 runHook preInstall
 mkdir $out
 cp -r dist/* $out
 runHook postInstall
 '';
 };
 # ...
}

This is a relatively simple derivation, mostly thanks to the magic of the buildNpmPackage function. This helper function takes care of creating an offline cache containing all of the NodeJS dependencies required to build the frontend. Nix builds are always done in an offline environment to help ensure reproducibility, so source code and dependencies are fetched (and hashed) early in the process before any software is actually built.

I chose to override the build phase to match the process used by the upstream Makefile. The result of this derivation is a package named scrutiny-webapp-frontend, which contains just the built output from the npm run build:prod command.

Next up was the dashboard backend. Another pleasingly simple derivation thanks to some help from buildGoModule:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


pkgs.buildGoModule rec {
 # The vendor hash is used for multiple packages, and
 # thus centralised and imported here.
 inherit version vendorHash;
 pname = "${name}-webapp-backend";

 # Use the source block defined in 'common.nix'
 src = repo;

 CGO_ENABLED = 0;

 # Override the build phase to ensure the correct
 # binary is built. This project ships both Go applications
 # and the NodeJS frontend in the same repository, so some
 # specificity is required.
 buildPhase = ''
 runHook preBuild
 go build \
 -o scrutiny-web \
 -ldflags="-extldflags=-static" \
 -tags "static netgo" \
 ./webapp/backend/cmd/scrutiny
 runHook postBuild
 '';

 # Add the built binary to the package output, as well
 # as the build output from the frontend.
 installPhase = ''
 mkdir -p $out/bin $out/share/scrutiny
 cp scrutiny-web $out/bin/scrutiny-web
 cp -r ${frontend}/* $out/share/scrutiny
 '';
}

This one is a little more interesting. There are some commonalities, such as setting the vendorHash to ensure the correct Go dependencies are used (imported from common.nix in this case), and overriding the build to match the upstream process and ensure the right binary is built.

Where things differ is in the install phase, where the contents of the scrutiny-webapp-frontend derivation is copied into the output of this derivation - which will ultimately result in a single Nix package (named scrutiny-webapp-backend) which will contain both the frontend and backend components of the application. This is ultimately exposed in an overlay as a package simply named scrutiny.

You can see the finished product (with additional package metadata) in app.nix and common.nix on Github.

Packaging the Collector #

The collector is just a single, statically compiled Go binary, and as such the derivation very much resembles that of the web application backend above:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39


{ pkgs, lib, ... }:
let
 # Again import common attributes from 'common.nix'
 common = import ./common.nix { inherit pkgs; };
 inherit (common) name version repo vendorHash;
in
pkgs.buildGoModule rec {
 inherit version vendorHash;
 pname = "${name}-collector";
 src = repo;

 # We'll create a wrapper script for the
 # collector, which makeWrapper helps with.
 buildInputs = with pkgs; [ makeWrapper ];

 CGO_ENABLED = 0;

 # Override the build phase to ensure the correct
 # binary is built.
 buildPhase = ''
 runHook preBuild
 go build \
 -o scrutiny-collector-metrics \
 -ldflags="-extldflags=-static" \
 -tags "static netgo" \
 ./collector/cmd/collector-metrics
 runHook postBuild
 '';

 # Install both the binary, and a generated wrapper script
 # which ensures that 'smartctl' is in the PATH of the collector.
 installPhase = ''
 mkdir -p $out/bin
 cp scrutiny-collector-metrics $out/bin/scrutiny-collector-metrics

 wrapProgram $out/bin/scrutiny-collector-metrics \
 --prefix PATH : ${lib.makeBinPath [ pkgs.smartmontools ]}
 '';
}

Of interest here is the installPhase. The collector works by invoking smartctl to scrape information from smartd. Scrutiny itself expects that tool to be readily available in it’s PATH, and to accomplish that I used the makeWrapper package to create a wrapper script that ensures scrutiny-collector-metrics is executed with the PATH correctly set such that smartctl can be found.

The final derivation for the collector can be seen in collector.nix on Github.

Writing a NixOS Module #

While I now had functioning packages for all of Scrutiny’s components, for them to function correctly on a machine there are a few things that need to be in place:

The dashboard package must be installed and started
The collector collector package must be installed and started
InfluxDB must be installed and started
Scrutiny dashboard must be configured to speak to the host’s InfluxDB
smartd must be installed and running

This sort of challenge is exactly what the NixOS modules system aims to solve. Modules take a set of configuration attributes, and convert them into rendered system configuration in the form of systemd units, configuration files and more.

It’s tempting to try to support all possible configurations when writing a module, but I generally prefer to start small, and support only the configuration I need for my use-case. This keeps things easy to test (and leaves some fun for the future!). In this case, the module would be first installed on a server which hosts a set of services behind the Traefik reverse proxy. To work out what configuration I wanted to provide, I looked at the upstream’s example config file. The important things that stood out to me for consideration were:

The location of the web UI files to serve
The host/port of the InfluxDB instance
The “base path” - Scrutiny will be exposed at https://<tailscale node>/scrutiny

As mentioned before - each NixOS module consists of some options, and some rendered config. The options block for my Scrutiny web app looks like the below snippet. I’ve omitted comments this time, as I think the language is quite descriptive here:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48


{
 let
 cfg = config.services.scrutiny;
 in
 options = {
 services.scrutiny = {
 enable = lib.mkEnableOption "Enables the scrutiny web application";

 # Use the `scrutiny` package by default.
 package = lib.mkPackageOptionMD pkgs "scrutiny" { };

 port = lib.mkOption {
 type = lib.types.port;
 default = 8080;
 description = lib.mdDoc "Port for web application to listen on";
 };

 host = lib.mkOption {
 type = lib.types.str;
 default = "0.0.0.0";
 description = lib.mdDoc "Interface address for web application to bind to";
 };

 basepath = lib.mkOption {
 type = lib.types.str;
 default = "";
 description = lib.mdDoc ''
 If Scrutiny will be behind a path prefixed reverse proxy, you can override this
 value to serve Scrutiny on a subpath.

 Do not include the leading '/'.
 '';
 };

 openFirewall = lib.mkOption {
 type = lib.types.bool;
 default = false;
 description = "Open the default ports in the firewall for Scrutiny.";
 };

 logLevel = lib.mkOption {
 type = lib.types.enum [ "INFO" "DEBUG" ];
 default = "INFO";
 description = lib.mdDoc "Log level for Scrutiny.";
 };
 };
 };
}

This provides some basic configuration for the attributes I care about - noticeably missing is any advanced configuration for InfluxDB (such as org, bucket, token), and and any ability to configure notifications which Scrutiny supports through the excellent shoutrrr library. These things will come later.

I needed a convenient way to convert this Nix-native configuration format into the right format for Scrutiny - I wrote a small Nix function to help with that, which takes configuration elements from the options defined above, and writes them into a small YAML file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


{
 mkScrutinyCfg = cfg: pkgs.writeTextFile {
 name = "scrutiny.yaml";
 text = ''
 version: 1
 web:
 listen:
 port: ${builtins.toString cfg.port}
 host: "${cfg.host}"
 basepath: "${if cfg.basepath != "" then "/" + cfg.basepath else ""}"

 database:
 location: "/var/lib/scrutiny/scrutiny.db"

 src:
 frontend:
 path: "${pkgs.scrutiny}/share/scrutiny"

 log:
 level: "${cfg.logLevel}"
 '';
 };
}

Note that this hard-codes some key elements, such as the path of the web application assets which are shipped as part of the scrutiny package.

Let’s take a look at the part of the module which turns these options into a rendered system configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


{
 config = {
 # If scrutiny is enabled, also enable InfluxDB with default settings
 services.influxdb2 = lib.mkIf cfg.enable {
 enable = true;
 };

 # Open the relevant ports in the system firewall if configured
 networking.firewall = lib.mkIf cfg.openFirewall {
 allowedTCPPorts = [ cfg.port ];
 };

 # If Scrutiny is enabled, create a systemd unit to start it
 systemd = {
 services = {
 scrutiny = lib.mkIf cfg.enable {
 description = "Hard Drive S.M.A.R.T Monitoring, Historical Trends & Real World Failure Thresholds";
 wantedBy = [ "multi-user.target" ];
 after = [ "network.target" ];
 serviceConfig = {
 # Don't run the application a root - create a dynamic user as it starts
 DynamicUser = true;
 # Start the application with a config rendered by the helper function
 ExecStart = "${cfg.package}/bin/scrutiny-web start --config ${mkScrutinyCfg cfg}";
 Restart = "always";
 StateDirectory = "scrutiny";
 StateDirectoryMode = "0750";
 };
 };
 };
 };
 };
}

That’s enough to get the dashboard started, but it doesn’t take care of starting the collector. For that, I added a couple more configuration options to the module:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


{
 let
 collectorCfg = config.services.scrutiny.collector;
 in
 {
 options = {
 services.scrutiny = {
 # ...
 collector = {
 enable = lib.mkOption {
 type = lib.types.bool;
 default = cfg.enable;
 description = lib.mdDoc "Enables the scrutiny collector";
 };

 # Use the `scrutiny-collector` package by default.
 package = lib.mkPackageOptionMD pkgs "scrutiny-collector" { };

 endpoint = lib.mkOption {
 type = lib.types.str;
 default = "http://${cfg.host}:${builtins.toString cfg.port}/${cfg.basepath}";
 description = lib.mdDoc "Scrutiny app API endpoint for sending metrics to.";
 };

 interval = lib.mkOption {
 type = lib.types.str;
 default = "15m";
 description = lib.mdDoc ''
 Interval on which to collect information about disks.

 Examples: 15m, 10s, 2h.
 '';
 };
 };
 };
 };
 }
}

With that configuration in place, I needed to adjust the rendered configuration to include starting the collector. The collector is designed to run on an interval to post metrics to the dashboard. The upstream achieves this with cron, but I decided to use systemd timers:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39


{
 config = {
 # ...
 # If the collector is enabled, enable smartd so that the
 # collector can scrape metrics from it
 services.smartd = {
 enable = collectorCfg.enable;
 extraOptions = [ "-A /var/log/smartd/" "--interval=600" ];
 };

 systemd = {
 services = {
 # Setup a systemd unit to start the collector
 scrutiny-collector = lib.mkIf collectorCfg.enable {
 description = "Scrutiny Collector Service";
 environment = {
 COLLECTOR_API_ENDPOINT = "${collectorCfg.endpoint}";
 };
 serviceConfig = {
 Type = "oneshot";
 ExecStart = "${cfg.collector.package}/bin/scrutiny-collector-metrics run";
 };
 };
 };

 # Set up a systemd timer to trigger the collector at an
 # interval (default 15m).
 timers = {
 scrutiny-collector = lib.mkIf collectorCfg.enable {
 timerConfig = {
 OnBootSec = "5m";
 OnUnitActiveSec = "${collectorCfg.interval}";
 Unit = "scrutiny-collector.service";
 };
 };
 };
 };
 };
}

And that’s it! You can see the final module all stitched together on Github.

Automated Testing #

To me, a super exciting part of the NixOS ecosystem is its automated testing framework, which is used to great effect for testing hundreds of packages end-to-end before they’re released to the various NixOS channels. The NixOS testing framework provides a set of helpers for spawning fresh NixOS virtual machines, and ensuring that applications can start, services have the right side effects, etc.

I wanted to write a simple test to validate that Scrutiny will continue to work as I update my flake. In my view, the test needed to:

Ensure that the packages could be built
Ensure that when services.scrutiny.enable = true is set, the services start
Ensure that the dashboard app renders the UI
Ensure that the metrics collector can speak to the dashboard

Most of these are relatively trivial - one piece that’s a little harder is testing that the UI renders correctly. The application is rendered using Javascript client-side, so a simple curl won’t get us the results we’re expecting. I had previously used selenium for this purpose when I submitted a test for the LXD UI, so I chose to use that approach again.

The test code can be seen below, or on Github:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69


{
 name = "scrutiny";
 nodes = {
 # Configure a NixOS virtual machine for the test
 machine = { self, pkgs, lib, ... }: {
 # Ensure that my NixOS module is imported
 imports = [ self.nixosModules.scrutiny ];
 # Add the package overlay from my flake so that `pkgs.scrutiny` resolves.
 nixpkgs.overlays = [ self.outputs.overlays.additions ];

 # Configure the VM to use my module and enable it
 services.scrutiny.enable = true;


 environment.systemPackages =
 let
 # A selenium script that fetches the dashboard using geckodriver
 seleniumScript = pkgs.writers.writePython3Bin "selenium-script"
 {
 libraries = with pkgs.python3Packages; [ selenium ];
 } ''
 from selenium import webdriver
 from selenium.webdriver.common.by import By
 from selenium.webdriver.firefox.options import Options
 from selenium.webdriver.support.ui import WebDriverWait

 options = Options()
 options.add_argument("--headless")
 service = webdriver.FirefoxService(executable_path="${lib.getExe pkgs.geckodriver}") # noqa: E501

 driver = webdriver.Firefox(options=options, service=service)
 driver.implicitly_wait(10)
 driver.get("http://localhost:8080/web/dashboard")

 wait = WebDriverWait(driver, 60)

 # Look for some elements that should be rendered by the Javascript bundle in the UI
 assert len(driver.find_elements(By.CLASS_NAME, "mat-button-wrapper")) > 0
 assert len(driver.find_elements(By.CLASS_NAME, "top-bar")) > 0

 driver.close()
 '';
 in
 with pkgs; [ curl firefox-unwrapped geckodriver seleniumScript ];
 };
 };
 # This is the test code that will check if our service is running correctly:
 testScript = ''
 start_all()

 # Wait for InfluxDB to be available
 machine.wait_for_unit("influxdb2")
 machine.wait_for_open_port(8086)

 # Wait for Scrutiny to be available
 machine.wait_for_unit("scrutiny")
 machine.wait_for_open_port(8080)

 # Ensure the API responds as we expect
 output = machine.succeed("curl localhost:8080/api/health")
 assert output == '{"success":true}'

 # Start the collector service to send some metrics
 collect = machine.succeed("systemctl start scrutiny-collector.service")

 # Ensure the application is actually rendered by the Javascript
 machine.succeed("PYTHONUNBUFFERED=1 selenium-script")
 '';
}

This relatively short code snippet will take care of:

Building a dedicated virtual machine according to the machine spec
Starting the VM
Running the tests inside the VM (specified in `testScript)
Collecting the results

I ended up adding this check in a Github Actions workflow so that it’s run each time I make a change to my flake.

Try It! #

You can give this a go too! If you’ve got a NixOS machine, or perhaps a virtual machine you’ve been experimenting with, then you need only add the following to your flake:

1
2
3
4
5
6
7
8
9


{
 inputs = {
 nixpkgs-unstable.url = "github:nixos/nixpkgs/nixos-unstable";
 jnsgruk.url = "github:jnsgruk/nixos-config";
 jnsgruk.inputs.nixpkgs.follows = "nixpkgs-unstable";
 # ...
 }
 # ...
}

And in your NixOS machine configuration:

1
2
3
4
5
6


# Where 'inputs' is the inputs attribute of your flake
{ inputs, ... }: {
 imports = [ inputs.jnsgruk.nixosModules.scrutiny ];
 nixpkgs.overlays = [ inputs.jnsgruk.overlays.additions ];
 services.scrutiny.enable = true;
}

The next time you rebuild your system, you should be able to browse to http://localhost:8080 and see some hard drive metrics!

What’s Next? #

This was a pretty quick exercise - I’d estimate at about 3 hours in total. No doubt I will find some bugs, but already in my mind are the following future improvements:

Add some more tests that exercise more of the config options
Enable notifications configuration in the module
Submit the packages/module to the upstream or to nixpkgs

I also need to go and investigate this rather sad looking hard disk…

That’s all for now! Thanks for reading if you got this far!

Integration testing with NixOS in Github Actions

Sat, 10 Feb 2024 00:00:00 +0000

Introduction #

While in my own time I’ve tended toward NixOS over the past 18 months, in my day-to-day work for Canonical I’m required to interact with a fair few of our products - and particularly build tools.

I frequently need to use some combination of Snaps, Charms and Rocks. Each of these have their own “craft” build tools (snapcraft, charmcraft and rockcraft), which are distributed exclusively as Snap packages and thus a little tricky to consume from NixOS.

The Problem #

Packaging the tools for Nix was a little repetitive, but not particularly difficult. They’re all built with Python, and share a common set of libraries. Testing that the packages were working correctly (i.e. could actually build software) on NixOS using the NixOS version of LXD in Github Actions proved more difficult.

Github Actions defaults to Ubuntu as the operating system for its runners - an entirely sensible choice, but not one that was going to help me test packages could work together on NixOS.

I could have hosted my own Github Actions runners to solve the problem, but I didn’t want to maintain such a deployment.

For a while I relied on just testing each of the crafts locally before pushing, and the CI simply installed the Nix package manager on the runners (using the excellent Nix installer from Determinate Systems) and ensured that the build could succeed, but this left a lot to be desired - particularly when I accidentally (and somewhat inevitably) broke one of the packages.

KVM for Github Actions #

Some time later I came across this post on the Github Blog, stating the following:

Starting on February 23, 2023, Actions users […] will be able to make use of hardware acceleration […].

What follows is an example of a relatively simple addition to a Github Workflow to enable KVM on Github Actions runners:

1
2
3
4
5


- name: Enable KVM group perms
 run: |
 echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules
 sudo udevadm control --reload-rules
 sudo udevadm trigger --name-match=kvm

Given the ability to relatively easily create NixOS VMs from a machine configuration, this should enable me to run a NixOS VM inside my Github Actions runners, and use that VM to run end to end tests of my craft packages.

After some quick tests, I confirmed that the above snippet worked just fine on the freely available runners that are assigned to public projects. After tooting excitedly about this, it was also picked up by the folks at Determinate Systems who promptly added support for this in their Nix install Github Action - enabling the feature by default.

Building VMs with Nix #

A really nice feature of NixOS that I discovered relatively late, is that given a NixOS machine configuration it’s trivial to build a virtual machine image for that configuration. This has the nice property that one can actually boot a VM-equivalent of any previously defined machines. You could, for example, boot a VM-equivalent of my laptop with the following command:

1

nix run github:jnsgruk/nixos-config#nixosConfigurations.freyja.config.system.build.vm

In order to test my craft tools, I needed a relatively simple NixOS VM that had LXD enabled, and my craft tools installed. My test VM configuration looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45


{ modulesPath, flake, pkgs, ... }: {
 # A nice helper that handles creating the VM launch script, which in turn
 # ensures the disk image is created as required, and QEMU is launched
 # with sensible parameters.
 imports = [ "${modulesPath}/virtualisation/qemu-vm.nix" ];

 # Define the version of NixOS and the architecture.
 system.stateVersion = "23.11";
 nixpkgs.hostPlatform = "x86_64-linux";

 # This overlay is provided by the crafts-flake, and ensures that
 # 'pkgs.snapcraft', 'pkgs.charmcraft', 'pkgs.rockcraft' all resolve to
 # the packages in the flake.
 nixpkgs.overlays = [ flake.outputs.overlay ];

 # These values are tuned such that the VM performs on Github Actions runners.
 virtualisation = {
 forwardPorts = [{ from = "host"; host.port = 2222; guest.port = 22; }];
 cores = 2;
 memorySize = 5120;
 diskSize = 10240;
 };

 # Configure the root user without password and enable SSH.
 # This VM will only ever be used in short-lived testing environments with
 # no inbound networking permitted, so there is minimal (if any) risk.
 # If you put this VM on the internet, you can keep the pieces! :)
 networking.firewall.enable = false;
 services.openssh.enable = true;
 services.openssh.settings.PermitRootLogin = "yes";
 users.extraUsers.root.password = "password";

 # Ensure that LXD is installed, and started on boot.
 virtualisation.lxd.enable = true;

 # Include the `craft-test` script, ensuring the craft apps are installed
 # and included in its PATH.
 environment.systemPackages = [
 (pkgs.writeShellApplication {
 name = "craft-test";
 runtimeInputs = with pkgs; [ unixtools.xxd git snapcraft charmcraft rockcraft ];
 text = builtins.readFile ./craft-test;
 })
 ];
}

Anybody can build and launch this VM trivially:

1

nix run github:jnsgruk/crafts-flake#testVM

Writing a Github workflow #

All the building blocks are in place! I wanted to keep the actual workflow definition for the tests as clean and understandable as possible, so I put together the craft-test script as a small helper which automates the building of real artefacts. An example invocation might be:

1

bash craft-test snapcraft

On each invocation, the script creates temporary directory, clones some representative build files for the selected craft tool, and launches the craft. The repos it uses for the representative packages are hard-coded for each craft for now.

I wrote one more small helper script to simplify connecting to the VM with the required parameters. It’s a wrapper around ssh and sshpass that’s hard-coded with the credentials of the test VM (don’t @ me!), and executes commands over SSH in the test VM. Using this script, one can bash vm-exec -- craft-test snapcraft and the craft-test script will be executed over SSH in the VM.

With all that said and done, the resulting workflow is pleasingly simple:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


# ...
jobs:
 test:
 runs-on: ubuntu-latest
 strategy:
 matrix:
 package: ["charmcraft", "rockcraft", "snapcraft"]
 steps:
 - name: Checkout flake
 uses: actions/checkout@v4

 - name: Install nix
 uses: DeterminateSystems/nix-installer-action@v9

 - name: Build and run the test VM
 run: |
 nix run .#testVm -- -daemonize -display none

 - name: Test ${{ matrix.package }}
 run: |
 nix run .#testVmExec -- craft-test ${{ matrix.package }}

A separate job is run for each of the crafts, and a real artefact is built in each, giving reasonable confidence that the consumers of my flake will be successful when building snaps, rocks and charms natively on NixOS. A successful run can be seen here.

Summary #

In this article we’ve covered:

Enabling KVM on Github Runners
Building NixOS VMs using Flakes
Booting NixOS VMs in Github Actions

If you’d like to build snaps, rocks or charms and you’re running NixOS, you can run the tools individually from my flake:

1
2
3
4
5
6
7
8


# Run charmcraft
nix run github:jnsgruk/crafts-flake#charmcraft

# Run rockcraft
nix run github:jnsgruk/crafts-flake#rockcraft

# Run snapcraft
nix run github:jnsgruk/crafts-flake#snapcraft

Or you can check out the README for instructions on how to integrate into your Nix config using overlays!

That’s all for now! 🤓

Github Actions on Jon Seager

Tracking Releases & CI Across Software Teams and Forges

Introduction #

Prior Art #

releasegen #

Parsing releasegen with Hugo #

Publishing with Github Actions #

Summary #

Simplifying Test & Release of Snapped GUI Apps

Introduction #

The Problem #

LXD Desktop VMs #

Wrapping LXD #

Building An Integrated Workflow #

Building A Screenshot Action #

End Result #

Summary #

Packaging Scrutiny for NixOS

Introduction #

Architecture #

Packaging for NixOS #

Packaging the Dashboard #

Packaging the Collector #

Writing a NixOS Module #

Automated Testing #

Try It! #

What’s Next? #

Integration testing with NixOS in Github Actions

Introduction #

The Problem #

KVM for Github Actions #

Building VMs with Nix #

Writing a Github workflow #

Summary #

`releasegen` #

Parsing `releasegen` with Hugo #