Nixpkgs on Jon Seager

Packaging the Multipass Flutter GUI for NixOS

Thu, 16 Jan 2025 00:00:00 +0000

Introduction #

The very first application I ever packaged in nixpkgs was Multipass. Multipass is, according to the website:

a CLI to launch and manage VMs on Windows, Mac and Linux that simulates a cloud environment with support for cloud-init.

I wrote in some detail about how I use Multipass and LXD on my workstation to create and manage VMs for testing and development. Last year Multipass shipped a new GUI written from the ground up in Flutter. It provides a clean, modern way to launch, manage and interact with VMs. When the GUI first shipped, I briefly attempted to get it to build with Nix, but some combination of my lack of knowledge, and the maturity of the Flutter tooling in nixpkgs at the time meant I never finished it.

As version 1.15.0 was in its release candidate stage, I decided to have another go!

This post will break down the process into steps, but if you’d like the tl;dr, take a look at the pull request.

Housekeeping #

Before getting started on packaging the GUI, I did some housekeeping on the Multipass package. First order of business was to simply bump the version to 1.15.0 and ensure the package still built, which it did.

Last year, RFC 140 was introduced to simplify the directory structure of nixpkgs, introducing a new pkgs/by-name directory which will (eventually) render pkgs/top-level/all-packages.nix useless. This new structure will make finding package definitions much easier, as they’ll all be arranged alphabetically by the first two characters of their name. Using the Multipass example, rather than pkgs/tools/virtualization/multipass/default.nix, the new preferred path would be pkgs/by-name/mu/multipass/package.nix. There was a talk at NixCon 2023 which summarised this nicely if you’d like more information.

As I was going to be carrying out some hefty work on Multipass by introducing the GUI, I took the opportunity to move to the new scheme as part of the changes.

Restructuring #

The Multipass package carries a few patches, and is already reasonably involved, so I decided that the best route forward was to create separate files for multipassd/multipass (the server & command line client binaries, written in C++, and already packaged) and multipass.gui (the new Flutter application). Thus the new structure for the package looks like so:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


.
# Derivation files
├── gui.nix
├── multipassd.nix
├── package.nix
# Lock file for Fluter app
├── pubspec.lock.json
# Update script for daemon/cli & GUI
├── update.sh
# Patches
├── cmake_no_fetch.patch
├── cmake_warning.patch
├── lxd_socket_path.patch
├── test_unreachable_call.patch
└── vcpkg_no_install.patch

Using `symlinkJoin` #

This package now takes a more unusual shape. Because the GUI is essentially a entirely separate application, written with a different toolkit in a different language, but I wanted the two to be installed as part of the overall multipass package, I decided to use two completely separate derivations and use symlinkJoin to bundle the two together. According to the manual, symlinkJoin:

can be used to put many derivations into the same directory structure. It works by creating a new derivation and adding symlinks to each of the paths listed. It expects two arguments, name, and paths. name (or alternatively pname and version) is the name used in the Nix store path for the created derivation. paths is a list of paths that will be symlinked. These paths can be to Nix store derivations or any other subdirectory contained within.

This is particularly useful for the Multipass package, because the Multipass GUI dynamically-links against libraries that are built as part of the multipassd derivation, and linking the two derivations in this way ensures that all those files are located correctly at runtime without any extra plumbing. This allowed me to keep the top-level package.nix relatively simple (annotated below):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55


{
# ... snip ...
}:
let
 name = "multipass";
 version = "1.15.0";

 # Fetch the source just once, to be used in both the multipassd and GUI derivations
 multipass_src = fetchFromGitHub {
 owner = "canonical";
 repo = "multipass";
 rev = "refs/tags/v${version}";
 hash = "sha256-RoOCh1winDs7BZwyduZziHj6EMe3sGMYonkA757UrIU=";
 fetchSubmodules = true;
 };

 # Shared metadata between derivations.
 commonMeta = {
 homepage = "https://multipass.run";
 license = lib.licenses.gpl3Plus;
 maintainers = with lib.maintainers; [ jnsgruk ];
 platforms = [ "x86_64-linux" ];
 };

 # Build the multipass server & client binaries from the shared source
 # and common metadata.
 multipassd = callPackage ./multipassd.nix {
 inherit commonMeta multipass_src version;
 };

 # Build the multipass GUI using shared source and common metadata.
 multipass-gui = callPackage ./gui.nix {
 inherit commonMeta multipass_src multipassd version;
 };
in
# Create the top-level `multipass` package using `symlinkJoin`
symlinkJoin {
 inherit version;
 pname = name;

 # Join both of the newly created derivations
 paths = [ multipassd multipass-gui ];

 # Ensure tests are run, and define a (new) update script.
 passthru = {
 tests = lib.optionalAttrs stdenv.hostPlatform.isLinux {
 inherit (nixosTests) multipass;
 };
 updateScript = ./update.sh;
 };

 meta = commonMeta // {
 description = "Ubuntu VMs on demand for any workstation";
 };
}

Building the GUI #

Normally the server, client and GUI binaries are all built as part of a single invocation of cmake. We can’t do that here, so the early code I put into the multipass package when I failed to package the GUI before still stood - essentially stopping cmake from trying to build the Flutter GUI:

1
2
3
4


{
 # We'll build the flutter application separately using buildFlutterApplication
 cmakeFlags = [ "-DMULTIPASS_ENABLE_FLUTTER_GUI=false" ];
}

Next, I created gui.nix which uses the pkgs.buildFlutterApplication helper. The derivation is annotated below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67


{
 # Passed in from the package.nix shown above
 commonMeta,
 multipass_src,
 multipassd,
 version,
 # ... snip ...
}:
flutter327.buildFlutterApplication {
 inherit version;
 pname = "multipass-gui";
 src = multipass_src;

 # Ensure we build in the correct directory for the GUI code.
 sourceRoot = "${multipass_src.name}/src/client/gui";

 # To make the builds as deterministic as possible, nixpkgs reads the pubspec.lock
 # file for the Flutter application and downloads/caches the correct versions of
 # flutter packages to be used in the build. It does this with `pub2nix`.
 #
 # Here I use a JSON representation of the pubspec.lock file committed into nixpkgs.
 pubspecLock = lib.importJSON ./pubspec.lock.json;

 # Some of the Flutter dependencies used are pulled from Github (branches) directly, so this
 # ensures that the versions/commits are pinned for future builds.
 gitHashes = {
 dartssh2 = "sha256-2pypKwurziwGLZYuGaxlS2lzN3UvJp3bRTvvYYxEqRI=";
 hotkey_manager_linux = "sha256-aO0h94YZvgV/ggVupNw8GjyZsnXrq3qTHRDtuhNv3oI=";
 system_info2 = "sha256-fly7E2vG+bQ/+QGzXk+DYba73RZccltdW2LpZGDKX60=";
 tray_menu = "sha256-riiAiBEms+9ARog8i+MR1fto1Yqx+gwbBWyNbNq6VTM=";
 window_size = "sha256-71PqQzf+qY23hTJvcm0Oye8tng3Asr42E2vfF1nBmVA=";
 xterm = "sha256-h8vIonTPUVnNqZPk/A4ZV7EYCMyM0rrErL9ZOMe4ZBE=";
 };

 # ... snip ...

 # The Multipass GUI relies on protobuf for API communication; ensure the
 # Dart protobuf definitions are compiled before we build.
 preBuild = ''
 mkdir -p lib/generated

 # Generate the Dart gRPC code for the Multipass GUI.
 protoc \
 --plugin=protoc-gen-dart=${lib.getExe protoc-gen-dart} \
 --dart_out=grpc:lib/generated \
 -I ../../rpc \
 ../../rpc/multipass.proto \
 google/protobuf/timestamp.proto
 '';

 # Libraries built by multipassd are linked dynamically at runtime; multipassd must
 # be installed for the GUI to function.
 runtimeDependencies = [ multipassd ];

 # Housekeeping to ensure the icons/desktop files are correctly installed.
 postFixup = ''
 mv $out/bin/multipass_gui $out/bin/multipass.gui

 install -Dm444 $out/app/multipass-gui/data/flutter_assets/assets/icon.png \
 $out/share/icons/hicolor/256x256/apps/multipass.gui.png

 cp $out/share/applications/multipass.gui.autostart.desktop \
 $out/share/applications/multipass.gui.desktop
 '';

 # ... snip ..
}

Update Script #

That was enough to ensure that the GUI worked correctly. My concern now lied with the maintenance of the package. The process for updating the derivation now involved:

Bumping the version string
Updating the source hashes
Updating the pubspec.lock.json file
Updating the gitHashes of the Flutter dependencies

This seemed a little convoluted and error prone to handle manually each time, so I decided to implement an update script for the package to take care of future version bumps. Thankfully nixpkgs has good provision for this, allowing package maintainers to specify passthru.updateScript to a derivation, which I did here.

I won’t post the full update script here, as it’s lots of unsightly bash! Essentially the script takes care of determining the latest version released on Github by using curl, then updates the pubspec.lock.json using a combination of curl/yj, then finally works through updating the source hash for the Multipass repo, gRPC fork it uses, and each of the git hashes for the Flutter dependencies.

This can be invoked from within the nixpkgs repo with:

1

nix-shell maintainers/scripts/update.nix --argstr package multipass

Summary #

Overall, this turned out cleaner than I expected, though it took me longer to figure out than this post might suggest. My hope is that this will a useful reference to anybody trying to package Flutter applications with Nix, and if you’re a user of the multipass package, check out the shiny new GUI that’s available to you! The GUI is available from 1.15.0 onwards, which at the time of writing means you’ll need to install the package from nixos-unstable or similar (sadly I didn’t land the change in time for NixOS 24.11).

In my opinion, the Multipass team have done a lovely job on the GUI - and for many users this will provide a much improved onboarding experience. Nice work 😎

A homelab dashboard for NixOS

Tue, 05 Mar 2024 00:00:00 +0000

Introduction #

I run a very small homelab that provides some basic services to my home network. I’m not much of a data hoarder, but my lab consists of some redundant storage in a raidz2 ZFS pool, and I use the homelab as a receive-only target for Syncthing, and as the point from which backups of my critical data are made using Borg & Borgbase.

It also runs a few other small services - all of which are exclusively available over Tailscale to my other devices. I wanted a small dashboard solution that could give me links to each of those services with a nice simple URL.

There are certainly plenty of options; this seems to be a highly crowded space in the open source homelab world. I settled on the rather ambiguously named homepage. At the time of writing, my dashboard looks like so, though there are people who have been far more creative with the appearance!

Naturally, I wanted to run this on NixOS, so in July 2023 I landed one of my early contributions to the project in the form of PR #243094 which added the package (named homepage-dashboard), a basic NixOS module and a basic test.

Homepage’s Configuration #

Homepage is configured using a set of YAML files named services.yaml, bookmarks.yaml, widgets.yaml, etc. When I originally started writing the module, it would look for those config files in a hard-coded location (/config), and if the files were missing it would copy a skeleton config into place with some defaults to get you going.

The hard-coded location results from the fact that the upstream primarily support deploying Homepage using Docker. They expect the config directory to be bind-mounted into the container from the host. As part of the initial packaging effort, I contributed a patch upstream to allow customising this location by setting the HOMEPAGE_CONFIG_DIR environment variable, which I then set in the systemd unit configuration in the NixOS module to /var/lib/homepage-dashboard.

This has been working fine for a few months, but it’s been bugging me that my dashboard configuration is not part of the declarative system configuration. Once the initial skeleton had been copied in place, you were left to edit the files manually (and back them up) if you wanted to make changes. Moreover, homepage defaults to creating a logs directory as a subdirectory of the config directory. This makes some sense in a container environment, but given that homepage also logs to stdout (which is collected by the systemd journal on NixOS), it’s really just unnecessary duplication.

Evolving The Module Design #

Before writing the actual implementation of the module, I decided to first sketch out what I wanted my NixOS configuration to look like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


{
 services.homepage-dashboard = {
 # These options were already present in my configuration.
 enable = true;
 package = unstable.homepage-dashboard;

 # The following options were what I planned to add.

 # https://gethomepage.dev/latest/configs/settings/
 settings = {};

 # https://gethomepage.dev/latest/configs/bookmarks/
 bookmarks = [];

 # https://gethomepage.dev/latest/configs/services/
 services = [];

 # https://gethomepage.dev/latest/configs/service-widgets/
 widgets = [];

 # https://gethomepage.dev/latest/configs/kubernetes/
 kubernetes = { };

 # https://gethomepage.dev/latest/configs/docker/
 docker = { };

 # https://gethomepage.dev/latest/configs/custom-css-js/
 customJS = "";
 customCSS = "";
 };
}

Each of the new sections would then map neatly to the different configuration section in the upstream documentation. Based on my learning from the Scrutiny module, I wanted to utilise the same RFC42 approach which would obviate the need for the module to specify every possible supported configuration option, resulting in a large, difficult to maintain module which could quickly fall behind the upstream project.

Homepage supports a large number of widgets (see below) which are able to scrape information from the API of various devices and services. These often require an API key or token of some kind, and having those in plaintext as part of the machine configuration is undesirable from a security perspective - even if your services are all on a private network like mine. Luckily I found out (tucked away in the docs) that Homepage can inject secret values into the configuration using environment variables.

Given the template value of {{HOMEPAGE_VAR_FOOBAR}} as part of the configuration, Homepage will automatically substitute the value of the variable HOMEPAGE_VAR_FOOBAR.

I decided to provide a single configuration option named environmentFile so that users can supply the path to an environment file containing all of their variables. This file can be omitted from Git repositories and configurations, or included in encrypted form. I achieve this by including the file encrypted using agenix which integrates @FiloSottile’s wonderful age into NixOS. You can see how that’s supplied as part of my nixos-config.

Backwards Compatibility #

According to my relatively naive Github search I estimated that there are not that many users of the module - likely in the tens, rather than the hundreds or thousands. That said, I think its important not to break those users. There’s no reason to expect that a nix flake update should break your system.

The way I chose to handle this in the module was to check if any of the new config options are set. If they’re not, the module behaves as before, but displays a deprecation warning:

The implementation of this check is relatively crude, but it works, and it will only be around until the release of NixOS 24.05 (in May 24):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


config =
 let
 # If homepage-dashboard is enabled, but none of the configuration values have been updated,
 # then default to "unmanaged" configuration which is manually updated in
 # var/lib/homepage-dashboard. This is to maintain backwards compatibility, and should be
 # deprecated in a future release.
 managedConfig = !(
 cfg.bookmarks == [ ] &&
 cfg.customCSS == "" &&
 cfg.customJS == "" &&
 cfg.docker == { } &&
 cfg.kubernetes == { } &&
 cfg.services == [ ] &&
 cfg.settings == { } &&
 cfg.widgets == [ ]
 );

 configDir = if managedConfig then "/etc/homepage-dashboard" else "/var/lib/homepage-dashboard";

 msg = "using unmanaged configuration for homepage-dashboard is deprecated and will be removed"
 + " in 24.05. please see the NixOS documentation for `services.homepage-dashboard' and add"
 + " your bookmarks, services, widgets, and other configuration using the options provided.";
 in
 lib.mkIf cfg.enable {
 # Display the deprecation warning if the configuration isn't managed
 warnings = lib.optional (!managedConfig) msg;

# ...

Solving Log Duplication #

I mentioned in a previous section that Homepage logs to both stdout and a logs directory by default. While the log file path can be customised, it’s not currently possible to disable the file logging completely. It’s not desirable to have the log file in this context, because all of the logs are collected by systemd anyway.

Looking at the upstream implementation, the logger is instantiated and configured in a single logger.js file. Homepage has a policy that they won’t accept feature contributions (even if you do the implementation) unless the feature gets at least 10 upvotes. I filed a feature request, but it’s yet to get enough votes to be accepted.

In the mean time I wrote a short patch on a branch in my personal fork which makes Homepage adhere to an environment variable named LOG_TARGETS. The possible values are both, file or stdout with a default value of both to respect the existing behaviour and remain backward compatible. The patch is now applied in the Nix package as part of nixpkgs, and the module configures the systemd unit by setting the LOG_TARGETS variable to stdout in cases where the configuration is managed:

1
2
3
4
5
6
7
8
9


{
 # ...
 environment = {
 HOMEPAGE_CONFIG_DIR = configDir;
 PORT = toString cfg.listenPort;
 LOG_TARGETS = lib.mkIf managedConfig "stdout";
 };
 # ...
}

Update: My feature request got upvoted pretty quickly, and as a result my pull request was merged. This means that following the next release of Homepage, I’ll be able to drop the patch from the Nix package. Thanks to all those who upvoted it!

Bolstering The Test Suite #

When I originally implemented the tests for the module, they simply enabled the service and ensure that it responded on the specified port. I wanted to include some logic in the test that ensured the ability to detect when managed configuration should be used, and when the module should respect an existing implementation.

The NixOS test suite supports specifying multiple machines as part of a given test, so extending the previous implementation wasn’t particularly cumbersome. See below for the (annotated) implementation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49


import ./make-test-python.nix ({ lib, ... }: {
 name = "homepage-dashboard";
 meta.maintainers = with lib.maintainers; [ jnsgruk ];

 # Create a machine that uses the legacy module format,
 # where configuration is unmanaged by nix, and relies
 # upon YAML files.
 nodes.unmanaged_conf = { pkgs, ... }: {
 services.homepage-dashboard.enable = true;
 };

 # Create another machine that sets some simple
 # configuration using the new module system. This
 # doesn't need to be exhaustive, just enough to trigger
 # the condition that makes the module use managed config.
 nodes.managed_conf = { pkgs, ... }: {
 services.homepage-dashboard = {
 enable = true;
 settings.title = "custom";
 };
 };

 testScript = ''
 # Ensure the services are started on unmanaged machine,
 # and that the service responds to HTTP requests on the
 # expected port.
 unmanaged_conf.wait_for_unit("homepage-dashboard.service")
 unmanaged_conf.wait_for_open_port(8082)
 unmanaged_conf.succeed("curl --fail http://localhost:8082/")

 # Ensure that /etc/homepage-dashboard doesn't exist, and boilerplate
 # configs are copied into place in `/var/lib/homepage-dashboard`.
 # This validates the existing behaviour.
 unmanaged_conf.fail("test -d /etc/homepage-dashboard")
 unmanaged_conf.succeed("test -f /var/lib/private/homepage-dashboard/settings.yaml")

 # Ensure the services are started on managed machine,
 # and that the service responds to HTTP requests on the
 # expected port.
 managed_conf.wait_for_unit("homepage-dashboard.service")
 managed_conf.wait_for_open_port(8082)
 managed_conf.succeed("curl --fail http://localhost:8082/")

 # Ensure /etc/homepage-dashboard is created and unmanaged
 # conf location isn't present
 managed_conf.succeed("test -d /etc/homepage-dashboard")
 managed_conf.fail("test -f /var/lib/private/homepage-dashboard/settings.yaml")
 '';
})

This is by no means exhaustive, and I can certainly imagine increasing the coverage here at a later date, but it does at least give some confidence when working on the module that the two basic modes of operation are functioning correctly.

Migrating Existing Configurations #

If you have been using the module in its past form, you may be wondering what the easiest way to migrate to the new format is…

I made the shift using yaml2nix to convert my existing YAML configurations to Nix expressions, and then formatted the output using nixpkgs-fmt. For example, given the following settings.yaml (which came from my homelab before I moved over):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


---
# For configuration options and examples, please see:
# https://gethomepage.dev/en/configs/settings

title: sgrs dashboard
favicon: https://jnsgr.uk/favicon.ico
headerStyle: clean

layout:
 media:
 style: row
 columns: 3
 infra:
 style: row
 columns: 4
 machines:
 style: row
 columns: 4

You can do the following to get a Nix expression that can be assigned to services.homepage-dashboard.settings in your machine configuration, converting the YAML to a Nix expression:

 ~/temp
❯ nix run nixpkgs#yaml2nix settings.yaml
{ title = "sgrs dashboard"; favicon = "https://jnsgr.uk/favicon.ico"; headerStyle = "clean"; layout = { media = { style = "row"; columns = 3; }; infra = { style = "row"; columns = 4; }; machines = { style = "row"; columns = 4; }; }; }

With that output, you can insert a few line breaks and rely on nixpkgs-fmt to get everything lined up properly. You can see my complete dashboard configuration in Nix format as part of my nixos-config repository.

Summary #

The PR was merged earlier today, and will now need to trickle through the branches on its way to nixos-unstable. At the time of writing, it hasn’t quite made it there:

You can track for yourself on the nixpkgs tracker, but the time delay should give you a chance to migrate your configuration!

See Github for the full module implementation, package and tests.

Let me know if you’re using the module, or if you run into any issues! If you’re a fan of Homepage, then consider helping out with the project or sponsoring them on Github, and once again thank you to those who helped review and shape the module as part of this upgrade!

Nixpkgs on Jon Seager

Packaging the Multipass Flutter GUI for NixOS

Introduction #

Housekeeping #

Restructuring #

Using symlinkJoin #

Building the GUI #

Update Script #

Summary #

A homelab dashboard for NixOS

Introduction #

Homepage’s Configuration #

Evolving The Module Design #

Backwards Compatibility #

Solving Log Duplication #

Bolstering The Test Suite #

Migrating Existing Configurations #

Summary #

Using `symlinkJoin` #