Charms on Jon Seager

Integration testing with NixOS in Github Actions

Sat, 10 Feb 2024 00:00:00 +0000

Introduction #

While in my own time I’ve tended toward NixOS over the past 18 months, in my day-to-day work for Canonical I’m required to interact with a fair few of our products - and particularly build tools.

I frequently need to use some combination of Snaps, Charms and Rocks. Each of these have their own “craft” build tools (snapcraft, charmcraft and rockcraft), which are distributed exclusively as Snap packages and thus a little tricky to consume from NixOS.

The Problem #

Packaging the tools for Nix was a little repetitive, but not particularly difficult. They’re all built with Python, and share a common set of libraries. Testing that the packages were working correctly (i.e. could actually build software) on NixOS using the NixOS version of LXD in Github Actions proved more difficult.

Github Actions defaults to Ubuntu as the operating system for its runners - an entirely sensible choice, but not one that was going to help me test packages could work together on NixOS.

I could have hosted my own Github Actions runners to solve the problem, but I didn’t want to maintain such a deployment.

For a while I relied on just testing each of the crafts locally before pushing, and the CI simply installed the Nix package manager on the runners (using the excellent Nix installer from Determinate Systems) and ensured that the build could succeed, but this left a lot to be desired - particularly when I accidentally (and somewhat inevitably) broke one of the packages.

KVM for Github Actions #

Some time later I came across this post on the Github Blog, stating the following:

Starting on February 23, 2023, Actions users […] will be able to make use of hardware acceleration […].

What follows is an example of a relatively simple addition to a Github Workflow to enable KVM on Github Actions runners:

1
2
3
4
5


- name: Enable KVM group perms
 run: |
 echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules
 sudo udevadm control --reload-rules
 sudo udevadm trigger --name-match=kvm

Given the ability to relatively easily create NixOS VMs from a machine configuration, this should enable me to run a NixOS VM inside my Github Actions runners, and use that VM to run end to end tests of my craft packages.

After some quick tests, I confirmed that the above snippet worked just fine on the freely available runners that are assigned to public projects. After tooting excitedly about this, it was also picked up by the folks at Determinate Systems who promptly added support for this in their Nix install Github Action - enabling the feature by default.

Building VMs with Nix #

A really nice feature of NixOS that I discovered relatively late, is that given a NixOS machine configuration it’s trivial to build a virtual machine image for that configuration. This has the nice property that one can actually boot a VM-equivalent of any previously defined machines. You could, for example, boot a VM-equivalent of my laptop with the following command:

1

nix run github:jnsgruk/nixos-config#nixosConfigurations.freyja.config.system.build.vm

In order to test my craft tools, I needed a relatively simple NixOS VM that had LXD enabled, and my craft tools installed. My test VM configuration looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45


{ modulesPath, flake, pkgs, ... }: {
 # A nice helper that handles creating the VM launch script, which in turn
 # ensures the disk image is created as required, and QEMU is launched
 # with sensible parameters.
 imports = [ "${modulesPath}/virtualisation/qemu-vm.nix" ];

 # Define the version of NixOS and the architecture.
 system.stateVersion = "23.11";
 nixpkgs.hostPlatform = "x86_64-linux";

 # This overlay is provided by the crafts-flake, and ensures that
 # 'pkgs.snapcraft', 'pkgs.charmcraft', 'pkgs.rockcraft' all resolve to
 # the packages in the flake.
 nixpkgs.overlays = [ flake.outputs.overlay ];

 # These values are tuned such that the VM performs on Github Actions runners.
 virtualisation = {
 forwardPorts = [{ from = "host"; host.port = 2222; guest.port = 22; }];
 cores = 2;
 memorySize = 5120;
 diskSize = 10240;
 };

 # Configure the root user without password and enable SSH.
 # This VM will only ever be used in short-lived testing environments with
 # no inbound networking permitted, so there is minimal (if any) risk.
 # If you put this VM on the internet, you can keep the pieces! :)
 networking.firewall.enable = false;
 services.openssh.enable = true;
 services.openssh.settings.PermitRootLogin = "yes";
 users.extraUsers.root.password = "password";

 # Ensure that LXD is installed, and started on boot.
 virtualisation.lxd.enable = true;

 # Include the `craft-test` script, ensuring the craft apps are installed
 # and included in its PATH.
 environment.systemPackages = [
 (pkgs.writeShellApplication {
 name = "craft-test";
 runtimeInputs = with pkgs; [ unixtools.xxd git snapcraft charmcraft rockcraft ];
 text = builtins.readFile ./craft-test;
 })
 ];
}

Anybody can build and launch this VM trivially:

1

nix run github:jnsgruk/crafts-flake#testVM

Writing a Github workflow #

All the building blocks are in place! I wanted to keep the actual workflow definition for the tests as clean and understandable as possible, so I put together the craft-test script as a small helper which automates the building of real artefacts. An example invocation might be:

1

bash craft-test snapcraft

On each invocation, the script creates temporary directory, clones some representative build files for the selected craft tool, and launches the craft. The repos it uses for the representative packages are hard-coded for each craft for now.

I wrote one more small helper script to simplify connecting to the VM with the required parameters. It’s a wrapper around ssh and sshpass that’s hard-coded with the credentials of the test VM (don’t @ me!), and executes commands over SSH in the test VM. Using this script, one can bash vm-exec -- craft-test snapcraft and the craft-test script will be executed over SSH in the VM.

With all that said and done, the resulting workflow is pleasingly simple:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


# ...
jobs:
 test:
 runs-on: ubuntu-latest
 strategy:
 matrix:
 package: ["charmcraft", "rockcraft", "snapcraft"]
 steps:
 - name: Checkout flake
 uses: actions/checkout@v4

 - name: Install nix
 uses: DeterminateSystems/nix-installer-action@v9

 - name: Build and run the test VM
 run: |
 nix run .#testVm -- -daemonize -display none

 - name: Test ${{ matrix.package }}
 run: |
 nix run .#testVmExec -- craft-test ${{ matrix.package }}

A separate job is run for each of the crafts, and a real artefact is built in each, giving reasonable confidence that the consumers of my flake will be successful when building snaps, rocks and charms natively on NixOS. A successful run can be seen here.

Summary #

In this article we’ve covered:

Enabling KVM on Github Runners
Building NixOS VMs using Flakes
Booting NixOS VMs in Github Actions

If you’d like to build snaps, rocks or charms and you’re running NixOS, you can run the tools individually from my flake:

1
2
3
4
5
6
7
8


# Run charmcraft
nix run github:jnsgruk/crafts-flake#charmcraft

# Run rockcraft
nix run github:jnsgruk/crafts-flake#rockcraft

# Run snapcraft
nix run github:jnsgruk/crafts-flake#snapcraft

Or you can check out the README for instructions on how to integrate into your Nix config using overlays!

That’s all for now! 🤓

Continuous Profiling for Juju with Parca

Tue, 02 Aug 2022 00:00:00 +0000

This post was originally posted on Charmhub on 02 August 2022. I’ve posted it to my blog retrospectively, but the article is unchanged. Some commands may need slight adjustments today, feel free to reach out if you get stuck!

Introduction #

Over the past few days, I’ve been diving into some charm development to get a hands-on feel for where Juju and the Charmed Operator ecosystem has gotten to over the past 15 months. In this post we will:

Outline the progress we’ve made, and the current state of charm development and integration
Show how the Observability libraries can be used to rapidly instrument charmed operators
Highlight a nice open source continuous profiling tool

But first, let’s start with a juicy screenshot showing where we’re headed with this post!

What is continuous profiling? #

Profiling itself is not a new concept. Profiling is all about measuring applications dynamically at runtime to understand their CPU usage, memory usage, and other important metrics. Profiling can yield deep insights about your application’s performance at runtime right down to the line of code that allocated some memory, or created a thread

Previously profiling was almost always performed on a “point in time” basis; continuous profiling improves upon this methodology by introducing a new dimension to your profiling results: time. The basic principle is that by profiling your code at regular intervals, you can understand more easily how different stimuli and runtime conditions affect the performance of your code, and its effect in turn on the underlying system.

Continuous profiling itself is not particularly new either, but its certainly a more regular topic of conversation for many in the last twelve months. Google wrote about it back in 2010 in a paper named Google Wide Profiling. They also stewarded the pprof project which not only defines a data format for profiles themselves, but resulted in the inclusion of the pprof Go package which provides a trivial means for including an HTTP server that’s ready to serve performance profiles in the pprof format, giving insights into memory allocations, CPU usage and goroutines created in the Go runtime.

Enter Parca #

Parca is an open source continuous profiling tool by Polar Signals. It’s written in Go, and takes inspiration from Prometheus in its configuration and storage of data. Better still, while Polar Signals are the primary stewards of Parca, the project is fully open source and has an open governance model which means anyone can become a maintainer.

We have been troubleshooting some crunchy performance issues in the Juju controller code of late, and it led me to think more carefully about profiling tools and start experimenting with Parca

Polar Signals distribute two key components for their solution: the Parca server, and the Parca agent. The server is responsible for collecting profiles across applications (and hosts) and making them available for viewing, filtering and analysis in a nice modern web interface. The agent is focused on specific hosts and unlocks some very powerful capabilities, including host-wide profiling using eBPF, which means you can benefit from profiling even if the applications themselves are not natively instrumented for profiling.

I read a little about Parca, and dived right in to configuring it in an SSH session on my controller (I even tweeted about it!). After I’d got things running, I decided to write a simple charm for Parca, but at the time it was just distributed by means of signed binaries in a Github release. I wanted an easier way to get access to Parca from within my charm that would save me the hassle of working out which architecture to fetch, verifying checksums and figuring out an escape hatch for “airgapped” systems. I decided to start building a snap package…

`snap install parca` #

Building a snap for Parca turned out to be a dream

Being written in Go gives it an unfair advantage, but at this early stage there isn’t a huge amount of config to contend with either. Even better, the server component requires very little access to the underlying host, making it a great candidate for a strictly confined snap. You can find the code for the Parca snap on Github. You get started right away with:

1

sudo snap install parca --channel edge

At the time of writing, the snap has two modes of operation: firstly it’ll make the parca command available for you to run one-shot. Parca is configured using command line flags in conjunction with a YAML file for specifying scrape targets and object storage. Because the snap is strictly confined, it can only read from /var/snap/parca/current and /var/snap/parca/common, so you’ll need to be careful about where you place your config file. The snap is also automatically granted access to /etc/parca, so you can drop the config file in /etc/parca/parca.yaml like the upstream docs suggest.

In addition to the “one-shot” mode of operation, the snap also provides a service. This essentially provides an out-of-the-box systemd configuration with a set of sensible defaults. Once installed you can start Parca immediately using the default configuration like so:

1
2
3
4


# Start the service
sudo snap start parca
# Follow the logs
sudo snap logs -f parca

There are a couple of configuration options, though they’re intentionally limited for now. If you look around carefully you’ll find options to configure the in-memory profile storage limit, and the experimental on-disk storage mode.

Charmed Parca #

With a nice convenient way to install and basically configure Parca on any Linux machine, it was time to craft a simple charm! I’ll not delve too deep here, as the details of writing Charmed Operators is well covered in the SDK Docs. I do want to highlight some interesting points from those early commits, though…

I created a simple class that represents Parca in the codebase. This is a very simple wrapper around a set of snap commands.
I’m utilising the snap library from Charmhub to keep things brief
The above class enables us to keep the actual charm class very simple indeed!

I used to opportunity to test out a new way of testing “machine charms”. If you’re so inclined, you can dig in and see that there are unit tests that focus solely on the lifecycle of the charm and its interation with the Juju hook tools, followed by functional tests which make no assumptions about being run in the context of Juju and validate only the system-level behaviour of installing and manipulating the snap, and finally integration tests which build and deploy the charm on a real LXD cluster.

But what about Kubernetes, I hear you ask? Well thankfully, because the kind folks at Polar Signals provide OCI images for each of their builds , it wasn’t much trouble to get it running on Kubernetes either, as early commits would indicate.

Integrating Parca with the Canonical Observability Stack #

Here’s where things got a bit magic Over the past 12 months, the Observability charm team have been busily working on the Canonical Observability Stack, which combines Prometheus, Grafana, Loki and Alertmanager into a turn-key observability solution on top of MicroK8s (or any other CNCF-conformant Kubernetes , for that matter.).

I mentioned earlier on that Parca took inspiration from Prometheus’ configuration when it comes to defining its profiling targets. This made my life very easy in this case. The Observability team have defined the prometheus_scrape relation which provides a Charm library that anyone can use to integrate their application such that prometheus can scrape it for metrics. In my case, I wanted to re-use the other side of that library! While the overall changes I made to the charm in this phase were large, this diff highlights they key changes I needed to make in order to reuse the existing implementation. I later forked this library and updated the interface name to parca_scrape to ensure that there are no mistakes when relating charms at runtime.

This meant that any charmed application could now trivially implement a relation that enabled it to be profiled by Parca . I chose to do this initially with a small stub operator called juju-introspect (Charmhub/Github).

Of course, once I’d worked this out, the implementation was relatively trivial to port to the Kubernetes charm too. Over a total of a few hours, I had created and published a Charmed Operator for Parca for both Kubernetes and machines that:

Installs Parca from the Snap store (machines only)
Starts and manages Parca’s configuration with minimal overhead on human operators
Enables Parca to profile other applications using relations
Provide integrations with Prometheus so that Parca can be scraped for metrics
Provide integrations with Grafana and include a default dashboard for both charms
Integrate with the Traefik ingress on Kubernetes

The other side of the coin is similarly approachable. Take the Zinc operator as an example - Zinc is a modern, efficient alternative to Elasticsearch written in Go, and happens to provide a profiling endpoint when configured correctly - you can see all of the effort required to enable that integration in the Juju model in this diff .

Try it out! #

So what if you want to get started?! I think there are two interesting experiments for the reader as a result of this article:

Profiling a Juju controller on LXD
Profiling Zinc on Kubernetes
Integrating Parca with the Canonical Observability Stack

The following sections assume that you have access to a machine with a LXD setup, and a MicroK8s setup. I use Multipass for this, but you can find a nice break down in the Juju docs if you need to get set up.

Profiling a Juju controller on LXD #

Make sure you’ve got access to a LXD cluster, and the perform the following steps to bootstrap a Juju controller, then augment the controller machine with the juju-introspect operator (I’m going to give this some more thought in coming months; as Juju 3.0 releases we’ll introduce a ‘controller charm’ that’ll be able to handle this natively!)

Check out the asciinema recording below for a detailed walkthrough, or follow the instructions below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# Bootstrap a new Juju controller on LXD
juju bootstrap localhost lxd

# Switch to the controller model
juju switch controller

# Deploy the charm to the controller machine
juju deploy juju-introspect -to 0 --channel edge

# Now you've got a controller machine up, you're ready to start profiling.
# Deploy the Parca operator
juju deploy parca --channel edge

# Relate Parca to the juju-introspect operator to configure a profiling target
juju relate parca juju-introspect

You can monitor the status of these operations with juju status, once things are settled you should be able to browse to the address of the Parca charm (on port 7070) and start exploring profiles.

Profiling Zinc on Kubernetes #

In this section, we’ll explore how to profile Zinc with Parca on Kubernetes with Juju, but also the integrations with Prometheus and Grafana. I’ve included a short recording here to show the process, and some abbreviated instructions…

Check out the asciinema recording below for a detailed walkthrough, or follow the instructions below:

Once you’re done with that, you can login to the Parca dashboard and start comparing profiles:

And also take time to check out the targets page, and notice that the target has been automatically labelled with information about the target application’s location in the Juju model:

To try out profiling Zinc with Parca yourself, try the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# Bootstrap a Juju controller on MicroK8s
juju bootstrap microk8s micro

# Add a model for us to deploy into
juju add-model dev

# Deploy Zinc
juju deploy zinc-k8s --trust

# Deploy Parca
juju deploy parca-k8s --channel edge --trust

# Relate the two apps to configure profiling
juju relate parca-k8s zinc-k8s:profiling-endpoint

As before, if you login to the Parca dashboard you should now see Zinc as a profiling target.

Integrating Parca with the Canonical Observability Stack #

As a bonus step, you can now deploy Prometheus and Grafana, and use them to monitor Parca itself!

Check out the asciinema recording below for a detailed walkthrough, or follow the instructions below. It assumes the starting point is the end-state of the Kubernetes walkthrough above:

Once you’re done, you can login to the Grafana instance and check the dashboard for Parca:

To try out the COS x Parca integration yourself, try the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


# Deploy Prometheus
juju deploy prometheus-k8s --trust --channel edge

# Deploy Grafana
juju deploy grafana-k8s --trust --channel edge

# Setup the relations
juju relate grafana-k8s:grafana-source prometheus-k8s
juju relate prometheus-k8s parca-k8s:metrics-endpoint
juju relate grafana-k8s parca-k8s

# Wait for things to settle in the Juju status output
# Get the admin password for Grafana
juju run-action grafana-k8s/0 get-admin-password --wait

# Now you should be able to login to the Grafana application at http://<app address>:3000
# and see the Parca dashboard!

And that’s it! You’re now profiling Zinc with Parca, which is being scraped by Prometheus for metrics, and dashboarded in Grafana.

Conclusions #

Well, that was fun! What have we learned?

The combination of snap packaging and Charmed Operators can make for a powerful, yet readable and maintainable set of automation for deploying applications anywhere
With the proliferation of more charm libraries (particularly from the Observability team!) it’s getting easier and easier for developers to integrate pieces of the Charmed Operator ecosystem
The Operator Framework makes it trivial to share code between operators designed for machines/bare-metal and Kubernetes, reducing the development time for operators
Parca looks really nice

I’m going to continue to play with Parca. I’d like to have a continuous profiling tool better integrated with Juju 3.0 so that the team, and others, can use it to help us improve the performance of Juju itself. With the introduction of the controller charm, this should be eminently feasible. We may seek one day to integrate a continuous profiling tool into the Canonical Observability Stack. If you’ve got ideas about this, or you’d like to contribute then get in touch and let’s talk about it!

Additionally, this post only really explored the possibility of using the Parca Server to scrape applications that are already set up for profiling. In the future I’d like to have a go at packaging the Parca Agent, so that we can unlock that juicy host-level, eBPF powered profiling and enable you get the benefit of continuous profiling in all of your deployments, irrespective of the technologies you’re deploying with Juju.

Links #

I’ve collected some links from throughout the article for reference:

Parca - Homepage / Github / Snap Store
Parca Operator - Github / Charmhub
Parca Kubernetes Operator - Github / Charmhub

Charms on Jon Seager

Integration testing with NixOS in Github Actions

Introduction #

The Problem #

KVM for Github Actions #

Building VMs with Nix #

Writing a Github workflow #

Summary #

Continuous Profiling for Juju with Parca

Introduction #

What is continuous profiling? #

Enter Parca #

snap install parca #

Charmed Parca #

Integrating Parca with the Canonical Observability Stack #

Try it out! #

Profiling a Juju controller on LXD #

Profiling Zinc on Kubernetes #

Integrating Parca with the Canonical Observability Stack #

Conclusions #

Links #

`snap install parca` #