Scrutiny on Jon Seager

Contributing Scrutiny to nixpkgs

Mon, 19 Feb 2024 00:00:00 +0000

Introduction #

I’m quite overwhelmed by the number of people who read, and engaged with my last post on packaging Scrutiny for NixOS. I hadn’t intended on posting to this blog more than maybe once per month, but I decided to continue documenting the journey of landing Scrutiny upstream in nixpkgs.

One of the things that really struck me about NixOS early on was how simple the contribution process is. A single Github repository contains all the packages, all the modules, and all the tests.

I’ve found the community incredibly helpful and respectful in all of my contributions - often with 2-3 people quickly providing reviews to help shape the contribution. I’ve heard complaints that reviews can be quite pedantic, and while I agree to some extent, the reviews nearly always come with concrete suggestions which are a great way to learn. Notwithstanding the fact that you are contributing to an operating system package that could land on many thousands of machines - it’s important to be careful!

If you’ve been deliberating about whether or not to submit your package, module or otherwise, I’d encourage you to go for it. I’ve always found it rewarding and it’s taught me a lot about Nix and NixOS.

My first contribution was adding a package, module and test for multipass - as you can see I had plenty to learn, but people were generous with their time and it resulted in a contribution that I use daily to get my work done on NixOS.

For those losing sleep over the poorly disk I showed in my last post, you’ll be relieved to know that it’s now been replaced! Incidentally this was the first time I’ve replaced a disk in a ZFS array and I was super impressed with how easy the process was!

Improving My Work #

Over the weekend following my first post about Scrutiny, a couple of people identified some small issues and made pull requests to resolve them - thank you to those people! You can see their work here, here and here. Someone also pointed out to me that string-interpolating YAML wasn’t the most ideal solution to rendering the configuration file, and the I could have used builtins.toJSON, taking advantage of the fact that JSON is also valid YAML.

I also wanted to make sure that there was a way for people to use the module with an existing InfluxDB deployment. My original implementation assumed that it could just add services.influxdb2.enable = true to the system configuration and use the default credentials to connect. I added some options to the module for configuring a different InfluxDB instance if required. These were later modified slightly to use lib.types.nullOr lib.types.str following a helpful review comment.

Next up was refactoring the helper function I wrote for generating the configuration file. As mentioned above, Nix comes with a handy builtins.toJSON function which takes a native Nix expression (an attribute set in this case) and renders it to JSON. This obviates the need to string-interpolate YAML, which is error prone at the best of times. The renewed function looked like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


{
 mkScrutinyCfg = cfg: pkgs.writeTextFile {
 name = "scrutiny.yaml";
 text = builtins.toJSON {
 version = 1;
 web = {
 listen = {
 inherit (cfg) host port;
 basepath = if cfg.basepath != "" then "/" + cfg.basepath else "";
 };

 database = {
 location = "/var/lib/scrutiny/scrutiny.db";
 };

 src = {
 frontend.path = "${pkgs.scrutiny}/share/scrutiny";
 };

 influxdb = {
 inherit (cfg.influxdb) scheme host port token org bucket;
 tls.insecure_skip_verify = cfg.influxdb.tlsSkipVerify;
 };
 };
 log = {
 level = cfg.logLevel;
 };
 };
 };
}

I also noticed that I had hard-coded the path to the binaries in the rendered systemd units. You might recall that each Nix package contains meta block, which adds information about the package such as the license, maintainer, etc. For each of the packages, I added a mainProgram attribute which enables the use of something like lib.getExe pkgs.scrutiny rather than ${pkgs.scrutiny}/bin/scrutiny.

1
2
3
4
5
6
7


{
 # ...
 meta = {
 mainProgram = "scrutiny";
 # ...
 };
}

Finally, I wanted to simplify the packaging of the Go programs slightly. In the previous post I used buildPhase to override the default build behaviour of the buildGoModule function. I later discovered the subPackages attribute, which enabled me to achieve the same effect without overriding the process manually. You can see the commit with all the changes on Github, but below is a (slightly modified) preview of the simplified collector derivation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


{ pkgs, lib, ... }:
pkgs.buildGoModule rec {
 # (source, name, version, vendor hash omitted)

 subPackages = "collector/cmd/collector-metrics/collector-metrics.go";

 buildInputs = with pkgs; [ makeWrapper ];

 CGO_ENABLED = 0;
 ldflags = [ "-extldflags=-static" ];
 tags = [ "netgo" "static" ];

 installPhase = ''
 mkdir -p $out/bin
 cp $GOPATH/bin/collector-metrics $out/bin/scrutiny-collector-metrics
 wrapProgram $out/bin/scrutiny-collector-metrics \
 --prefix PATH : ${lib.makeBinPath [ pkgs.smartmontools ]}
 '';

 # (meta block omitted)
}

Contributing To nixpkgs #

With these changes made, it was time to open a pull request. The NixOS community provide some contributing guidelines which are worth a read before you start.

There was a structural transition in the pkgs directory lately, so this was my first contribution under that new structure. There was a detailed talk about that transition at Nixcon if you’d like more information.

I needed to do the following:

Create the scrutiny and scrutiny-collector packages
Add the scrutiny module
Add the tests
Update the release notes for the next release of NixOS

Which translated into the following stack of commits on PR #289934:

There were a few other minor changes, such as how input packages (like smartmontools and makeWrapper) are passed to the packages. In the nixpkgs repository, packages are passed directly to the derivations, rather than pkgs being passed as a top-level argument and packages being referred to by pkgs.smartmontools etc.

I also learned something new about module configuration. In the first iteration I had set the default value of services.scrutiny.collector.enable to the value of services.scrutiny.enable, meaning that the collector would be enabled automatically when Scrutiny was enabled if no other configuration was specified. It transpires that this is not permitted in nixpkgs, and resulted in this CI failure. I updated the module with a subtle API change, meaning that consumers of the module will now need to do the following to get the same behaviour.

1
2
3
4
5
6


{
 services = {
 scrutiny.enable = true;
 scrutiny.collector.enable = true;
 };
}

Review Feedback #

From the first review I learned about the lib.types.nullOr meta-type which enables the specification of optionally null configuration options in a NixOS module.

What followed was a suggestion to embrace a different configuration format according to RFC42: a proposal for structured module configuration which is more rigorously type checked, and more flexible as new options are added to the underlying workload.

Many NixOS modules solve this problem today with options like extraConfig or configLines, which append strings to the end of any options-generated configuration file. What RFC42 proposes is a hybrid approach whereby certain options are well-defined (and thus feature in the documentation), but additional options can be injected as needed. The has the nice side-effect that modules do not need to support dozens of options which can become a burden to maintain and test over time, but consumers can still make use of new features and options that are introduced to the underlying workload.

You can see the full resulting options definition on Github, and small annotated excerpt below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


{
 services.scrutiny = {
 enable = lib.mkEnableOption "Enables the scrutiny web application.";

 # ...

 settings = lib.mkOption {
 description = lib.mdDoc ''
 Scrutiny settings to be rendered into the configuration file.

 See https://github.com/AnalogJ/scrutiny/blob/master/example.scrutiny.yaml.
 '';

 default = { };

 # This is the key to RFC42 - defining `settings` as type `submodule`.
 type = lib.types.submodule {
 # This ensures that attrsets, lists, etc. are rendered as JSON
 freeformType = (pkgs.formats.yaml { }).type;

 # The following are examples of "well-defined" options with
 # formal types and metadata.
 options.web.listen.port = lib.mkOption {
 type = lib.types.port;
 default = 8080;
 description = lib.mdDoc "Port for web application to listen on.";
 };

 options.web.listen.host = lib.mkOption {
 type = lib.types.str;
 default = "0.0.0.0";
 description = lib.mdDoc "Interface address for web application to bind to.";
 };

 # ...
 };
 };
}

This enables the following user configurations:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


{
 services.scrutiny = {
 enable = true;
 settings = {
 # These options are well-defined in the module with types,
 # examples, descriptions and are rendered in the docs.
 web.listen = {
 port = 8080;
 host = "100.64.12.34";
 };

 # These are not, but will be rendered still as JSON and enable
 # people to make use of options supported by Scrutiny, but
 # not directly by the module.
 notify = {
 urls = [
 "discord://token@channel"
 ];
 };
 };
 };
}

I was impressed by the simplicity of the approach, and subsequently adopted the same for the collector, which now has the ability to take configuration through the services.scrutiny.collector.settings attribute.

One aspect of Scrutiny’s design that helped is the ability to take configuration either through from file, or through environment variables, or both! I chose to supply some fixed options as environment variables to the systemd services, and allow the rest of the configuration to be set using the generated file.

@JulienMalka reviewed next, suggesting changes to ensure that various preInstall and postInstall hooks were being called where I was overriding installPhase, though what followed from @katexochen’s review was a simplification that reduced this need slightly.

@katexochen suggested a couple of nice changes, among which was the ability to drop explicitly mentioning the netgo tag in the Go packages, since CGO_ENABLED is disabled already, this implies using the Go internal version of many libraries. They also suggested moving the install of the frontend pieces to postInstall, rather than overriding the standard installPhase.

And finally @SuperSandro2000 reviewed with some naming changes to simplify the package definitions and reduce repetition. I also noticed that I was unnecessarily declaring the frontend definition using the rec keyword. Once these were fixed the PR was merged!

Contribution Tips #

I’m still relatively new to this - you can see that my list of contributions is not extensive, but I’ve been through the process of adding, updating and refining packages and modules a few times.

Take this advice for what it is, but I hope you find it helpful:

Read the contribution guidelines: This is probably the best start. Try to follow the guidance as best you can, and take a look at some other recently closed Pull Requests to see if you can pick up on any patterns.
Be humble: Everyone is new when they start. You might find some reviews a little terse. Rest assured that this is only because the maintainers are incredibly busy. Assume good intentions, remain polite and respectful. Nixpkgs is one of the busiest Github repositories I’ve seen, and it’s largely maintained by volunteers who often have limited time to give for a huge number of contributions.
Be patient: Don’t expect a review immediately. If you’re looking for a preliminary review, feel free to ping me on Mastodon and I’ll try to offer any guidance I can.
Use the Discourse: Because of the volume of PRs, the community run ongoing “PRs Ready for Review” threads on Discourse which you can use to attract reviewers if your PR has been stale a while. It’s also a great place to ask for help.

Once merged, commits trickle through the various branches of the repository. You can track the progress of your PR in that process once it’s merged using the PR Tracker. You can see an example of this by tracking the Scrutiny PR:

Summary & Thanks #

My Scrutiny packages, module and tests landed in nixpkgs approximately 4 days after I submitted the pull request. As ever, I received a bunch of helpful reviews from the community, and Scrutiny is now easier than ever to consume on NixOS.

You can see the finished components here:

If you were previously using my flake to consume Scrutiny, you’ll now see a deprecation warning. To migrate over, import the module from unstable in your flake and ensure that you explicitly enable the collector:

1
2
3
4
5
6


{
 services = {
 scrutiny.enable = true;
 scrutiny.collector.enable = true;
 };
}

Before I wrap up, I’d like to thank all those people who took the time to review the PR, and especially @JulienMalka and @RaitoBezarius who have helped me through countless contributions over the past months and been incredibly responsive to my questions.

Give it a go and let me know how you get on!

Packaging Scrutiny for NixOS

Fri, 16 Feb 2024 00:00:00 +0000

Update: Since writing this post, I’ve contributed Scrutiny upstream to nixpkgs/NixOS. You can see the write up of that effort here.

Introduction #

In a recent (well, recent-ish) episode of the Self Hosted Show, there was some talk of a hard drive monitoring tool called Scrutiny. Scrutiny is a hard drive monitoring tool that exposes S.M.A.R.T data in a nice, clean dashboard. It gathers that S.M.A.R.T data using the venerable smartd, which is a Linux daemon that monitors S.M.A.R.T data from a huge number of ATA, IDE, SCSI-3 drives. The code is available on Github.

The aim of running such monitoring is to detect and replace failing hard drives before they cause an outage, or any data loss. Depending on the firmware of a drive, there are potentially hundreds of S.M.A.R.T attributes that can be collected, so it can be hard to understand which to pay attention to.

As good as smartd is, it has some shortcomings such as not recording historical data for each attribute, and relying upon value thresholds that are set by the manufacturers of the drive. Scrutiny aims to provide historical tracking of attribute readings, and thresholds based on real-world data from Backblaze.

This all looked very compelling, but it hasn’t been packaged for NixOS as far as I can tell. This post is quite “code heavy” and is intended as a bit of a deep-dive/walkthrough for people who are new to packaging in Nix. I’m certainly no expert here, and if you spot something I’ve done wrong - I’d love to hear about it!

Below is a screenshot of Scrutiny’s dashboard:

Architecture #

In order to get Scrutiny up and running, there were a few components that all needed packaging separately.

Scrutiny itself is made up of the following:

A web application backend - written in Go
A web application frontend - written in NodeJs/Angular
A collector service - written in Go

The web application is the dashboard that you’ll see the pretty screenshots of. The collector service is designed to be run on an interval to collect information from smartd, and send it to the web application’s API.

Scrutiny relies upon InfluxDB to store data about smart attributes in a timeseries.

Thinking about how to structure things, I decided that I would build two separate Nix packages: one for the dashboard and UI, and another for the collector. It seems one could run the collector and dashboard on different machines, so this seemed like a logical split.

Packaging for NixOS #

Packaging the Dashboard #

I decided to start with the web application. Because of how the project is laid out, this means combining two separate derivations based built from the same source code: one for the UI and one for the backend. I began by creating a file that would carry common attributes for each of the derivations, such as the version, repository information, hash, etc.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


# common.nix
{ pkgs , ... }:
let
 name = "scrutiny";
 version = "0.7.2";
in
{
 inherit name version;

 # Fetch the source code from Github, referring to
 # the version required by Git tag.
 repo = pkgs.fetchFromGitHub {
 owner = "AnalogJ";
 repo = name;
 rev = "v${version}";
 sha256 = "sha256-UYKi+WTsasUaE6irzMAHr66k7wXyec8FXc8AWjEk0qs=";
 };

 # ...
}

I then started packaging the frontend:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36


{ pkgs, ... }: {
 # Import some common attributes such
 # as name, version, src repo
 common = import ./common.nix { inherit pkgs; };
 inherit (common) name version repo vendorHash;

 # Create a package for the Javascript frontend
 frontend = pkgs.buildNpmPackage rec {
 inherit version;
 pname = "${name}-webapp-frontend";
 src = "${repo}/webapp/frontend";

 # This hash is generated at build time, and uniquely identifies
 # the cache of NodeJS dependencies used for the build.
 npmDepsHash = "sha256-M8P41LPg7oJ/C9abDuNM5Mn+OO0zK56CKi2BwLxv8oQ=";

 # Override the build phase to match the
 # upstream build process.
 buildPhase = ''
 runHook preBuild
 mkdir dist
 npm run build:prod --offline -- --output-path=dist
 runHook postBuild
 '';

 # Copy the output of the compiled javascript bundle
 # and site assets to the package output.
 installPhase = ''
 runHook preInstall
 mkdir $out
 cp -r dist/* $out
 runHook postInstall
 '';
 };
 # ...
}

This is a relatively simple derivation, mostly thanks to the magic of the buildNpmPackage function. This helper function takes care of creating an offline cache containing all of the NodeJS dependencies required to build the frontend. Nix builds are always done in an offline environment to help ensure reproducibility, so source code and dependencies are fetched (and hashed) early in the process before any software is actually built.

I chose to override the build phase to match the process used by the upstream Makefile. The result of this derivation is a package named scrutiny-webapp-frontend, which contains just the built output from the npm run build:prod command.

Next up was the dashboard backend. Another pleasingly simple derivation thanks to some help from buildGoModule:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


pkgs.buildGoModule rec {
 # The vendor hash is used for multiple packages, and
 # thus centralised and imported here.
 inherit version vendorHash;
 pname = "${name}-webapp-backend";

 # Use the source block defined in 'common.nix'
 src = repo;

 CGO_ENABLED = 0;

 # Override the build phase to ensure the correct
 # binary is built. This project ships both Go applications
 # and the NodeJS frontend in the same repository, so some
 # specificity is required.
 buildPhase = ''
 runHook preBuild
 go build \
 -o scrutiny-web \
 -ldflags="-extldflags=-static" \
 -tags "static netgo" \
 ./webapp/backend/cmd/scrutiny
 runHook postBuild
 '';

 # Add the built binary to the package output, as well
 # as the build output from the frontend.
 installPhase = ''
 mkdir -p $out/bin $out/share/scrutiny
 cp scrutiny-web $out/bin/scrutiny-web
 cp -r ${frontend}/* $out/share/scrutiny
 '';
}

This one is a little more interesting. There are some commonalities, such as setting the vendorHash to ensure the correct Go dependencies are used (imported from common.nix in this case), and overriding the build to match the upstream process and ensure the right binary is built.

Where things differ is in the install phase, where the contents of the scrutiny-webapp-frontend derivation is copied into the output of this derivation - which will ultimately result in a single Nix package (named scrutiny-webapp-backend) which will contain both the frontend and backend components of the application. This is ultimately exposed in an overlay as a package simply named scrutiny.

You can see the finished product (with additional package metadata) in app.nix and common.nix on Github.

Packaging the Collector #

The collector is just a single, statically compiled Go binary, and as such the derivation very much resembles that of the web application backend above:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39


{ pkgs, lib, ... }:
let
 # Again import common attributes from 'common.nix'
 common = import ./common.nix { inherit pkgs; };
 inherit (common) name version repo vendorHash;
in
pkgs.buildGoModule rec {
 inherit version vendorHash;
 pname = "${name}-collector";
 src = repo;

 # We'll create a wrapper script for the
 # collector, which makeWrapper helps with.
 buildInputs = with pkgs; [ makeWrapper ];

 CGO_ENABLED = 0;

 # Override the build phase to ensure the correct
 # binary is built.
 buildPhase = ''
 runHook preBuild
 go build \
 -o scrutiny-collector-metrics \
 -ldflags="-extldflags=-static" \
 -tags "static netgo" \
 ./collector/cmd/collector-metrics
 runHook postBuild
 '';

 # Install both the binary, and a generated wrapper script
 # which ensures that 'smartctl' is in the PATH of the collector.
 installPhase = ''
 mkdir -p $out/bin
 cp scrutiny-collector-metrics $out/bin/scrutiny-collector-metrics

 wrapProgram $out/bin/scrutiny-collector-metrics \
 --prefix PATH : ${lib.makeBinPath [ pkgs.smartmontools ]}
 '';
}

Of interest here is the installPhase. The collector works by invoking smartctl to scrape information from smartd. Scrutiny itself expects that tool to be readily available in it’s PATH, and to accomplish that I used the makeWrapper package to create a wrapper script that ensures scrutiny-collector-metrics is executed with the PATH correctly set such that smartctl can be found.

The final derivation for the collector can be seen in collector.nix on Github.

Writing a NixOS Module #

While I now had functioning packages for all of Scrutiny’s components, for them to function correctly on a machine there are a few things that need to be in place:

The dashboard package must be installed and started
The collector collector package must be installed and started
InfluxDB must be installed and started
Scrutiny dashboard must be configured to speak to the host’s InfluxDB
smartd must be installed and running

This sort of challenge is exactly what the NixOS modules system aims to solve. Modules take a set of configuration attributes, and convert them into rendered system configuration in the form of systemd units, configuration files and more.

It’s tempting to try to support all possible configurations when writing a module, but I generally prefer to start small, and support only the configuration I need for my use-case. This keeps things easy to test (and leaves some fun for the future!). In this case, the module would be first installed on a server which hosts a set of services behind the Traefik reverse proxy. To work out what configuration I wanted to provide, I looked at the upstream’s example config file. The important things that stood out to me for consideration were:

The location of the web UI files to serve
The host/port of the InfluxDB instance
The “base path” - Scrutiny will be exposed at https://<tailscale node>/scrutiny

As mentioned before - each NixOS module consists of some options, and some rendered config. The options block for my Scrutiny web app looks like the below snippet. I’ve omitted comments this time, as I think the language is quite descriptive here:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48


{
 let
 cfg = config.services.scrutiny;
 in
 options = {
 services.scrutiny = {
 enable = lib.mkEnableOption "Enables the scrutiny web application";

 # Use the `scrutiny` package by default.
 package = lib.mkPackageOptionMD pkgs "scrutiny" { };

 port = lib.mkOption {
 type = lib.types.port;
 default = 8080;
 description = lib.mdDoc "Port for web application to listen on";
 };

 host = lib.mkOption {
 type = lib.types.str;
 default = "0.0.0.0";
 description = lib.mdDoc "Interface address for web application to bind to";
 };

 basepath = lib.mkOption {
 type = lib.types.str;
 default = "";
 description = lib.mdDoc ''
 If Scrutiny will be behind a path prefixed reverse proxy, you can override this
 value to serve Scrutiny on a subpath.

 Do not include the leading '/'.
 '';
 };

 openFirewall = lib.mkOption {
 type = lib.types.bool;
 default = false;
 description = "Open the default ports in the firewall for Scrutiny.";
 };

 logLevel = lib.mkOption {
 type = lib.types.enum [ "INFO" "DEBUG" ];
 default = "INFO";
 description = lib.mdDoc "Log level for Scrutiny.";
 };
 };
 };
}

This provides some basic configuration for the attributes I care about - noticeably missing is any advanced configuration for InfluxDB (such as org, bucket, token), and and any ability to configure notifications which Scrutiny supports through the excellent shoutrrr library. These things will come later.

I needed a convenient way to convert this Nix-native configuration format into the right format for Scrutiny - I wrote a small Nix function to help with that, which takes configuration elements from the options defined above, and writes them into a small YAML file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


{
 mkScrutinyCfg = cfg: pkgs.writeTextFile {
 name = "scrutiny.yaml";
 text = ''
 version: 1
 web:
 listen:
 port: ${builtins.toString cfg.port}
 host: "${cfg.host}"
 basepath: "${if cfg.basepath != "" then "/" + cfg.basepath else ""}"

 database:
 location: "/var/lib/scrutiny/scrutiny.db"

 src:
 frontend:
 path: "${pkgs.scrutiny}/share/scrutiny"

 log:
 level: "${cfg.logLevel}"
 '';
 };
}

Note that this hard-codes some key elements, such as the path of the web application assets which are shipped as part of the scrutiny package.

Let’s take a look at the part of the module which turns these options into a rendered system configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


{
 config = {
 # If scrutiny is enabled, also enable InfluxDB with default settings
 services.influxdb2 = lib.mkIf cfg.enable {
 enable = true;
 };

 # Open the relevant ports in the system firewall if configured
 networking.firewall = lib.mkIf cfg.openFirewall {
 allowedTCPPorts = [ cfg.port ];
 };

 # If Scrutiny is enabled, create a systemd unit to start it
 systemd = {
 services = {
 scrutiny = lib.mkIf cfg.enable {
 description = "Hard Drive S.M.A.R.T Monitoring, Historical Trends & Real World Failure Thresholds";
 wantedBy = [ "multi-user.target" ];
 after = [ "network.target" ];
 serviceConfig = {
 # Don't run the application a root - create a dynamic user as it starts
 DynamicUser = true;
 # Start the application with a config rendered by the helper function
 ExecStart = "${cfg.package}/bin/scrutiny-web start --config ${mkScrutinyCfg cfg}";
 Restart = "always";
 StateDirectory = "scrutiny";
 StateDirectoryMode = "0750";
 };
 };
 };
 };
 };
}

That’s enough to get the dashboard started, but it doesn’t take care of starting the collector. For that, I added a couple more configuration options to the module:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


{
 let
 collectorCfg = config.services.scrutiny.collector;
 in
 {
 options = {
 services.scrutiny = {
 # ...
 collector = {
 enable = lib.mkOption {
 type = lib.types.bool;
 default = cfg.enable;
 description = lib.mdDoc "Enables the scrutiny collector";
 };

 # Use the `scrutiny-collector` package by default.
 package = lib.mkPackageOptionMD pkgs "scrutiny-collector" { };

 endpoint = lib.mkOption {
 type = lib.types.str;
 default = "http://${cfg.host}:${builtins.toString cfg.port}/${cfg.basepath}";
 description = lib.mdDoc "Scrutiny app API endpoint for sending metrics to.";
 };

 interval = lib.mkOption {
 type = lib.types.str;
 default = "15m";
 description = lib.mdDoc ''
 Interval on which to collect information about disks.

 Examples: 15m, 10s, 2h.
 '';
 };
 };
 };
 };
 }
}

With that configuration in place, I needed to adjust the rendered configuration to include starting the collector. The collector is designed to run on an interval to post metrics to the dashboard. The upstream achieves this with cron, but I decided to use systemd timers:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39


{
 config = {
 # ...
 # If the collector is enabled, enable smartd so that the
 # collector can scrape metrics from it
 services.smartd = {
 enable = collectorCfg.enable;
 extraOptions = [ "-A /var/log/smartd/" "--interval=600" ];
 };

 systemd = {
 services = {
 # Setup a systemd unit to start the collector
 scrutiny-collector = lib.mkIf collectorCfg.enable {
 description = "Scrutiny Collector Service";
 environment = {
 COLLECTOR_API_ENDPOINT = "${collectorCfg.endpoint}";
 };
 serviceConfig = {
 Type = "oneshot";
 ExecStart = "${cfg.collector.package}/bin/scrutiny-collector-metrics run";
 };
 };
 };

 # Set up a systemd timer to trigger the collector at an
 # interval (default 15m).
 timers = {
 scrutiny-collector = lib.mkIf collectorCfg.enable {
 timerConfig = {
 OnBootSec = "5m";
 OnUnitActiveSec = "${collectorCfg.interval}";
 Unit = "scrutiny-collector.service";
 };
 };
 };
 };
 };
}

And that’s it! You can see the final module all stitched together on Github.

Automated Testing #

To me, a super exciting part of the NixOS ecosystem is its automated testing framework, which is used to great effect for testing hundreds of packages end-to-end before they’re released to the various NixOS channels. The NixOS testing framework provides a set of helpers for spawning fresh NixOS virtual machines, and ensuring that applications can start, services have the right side effects, etc.

I wanted to write a simple test to validate that Scrutiny will continue to work as I update my flake. In my view, the test needed to:

Ensure that the packages could be built
Ensure that when services.scrutiny.enable = true is set, the services start
Ensure that the dashboard app renders the UI
Ensure that the metrics collector can speak to the dashboard

Most of these are relatively trivial - one piece that’s a little harder is testing that the UI renders correctly. The application is rendered using Javascript client-side, so a simple curl won’t get us the results we’re expecting. I had previously used selenium for this purpose when I submitted a test for the LXD UI, so I chose to use that approach again.

The test code can be seen below, or on Github:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69


{
 name = "scrutiny";
 nodes = {
 # Configure a NixOS virtual machine for the test
 machine = { self, pkgs, lib, ... }: {
 # Ensure that my NixOS module is imported
 imports = [ self.nixosModules.scrutiny ];
 # Add the package overlay from my flake so that `pkgs.scrutiny` resolves.
 nixpkgs.overlays = [ self.outputs.overlays.additions ];

 # Configure the VM to use my module and enable it
 services.scrutiny.enable = true;


 environment.systemPackages =
 let
 # A selenium script that fetches the dashboard using geckodriver
 seleniumScript = pkgs.writers.writePython3Bin "selenium-script"
 {
 libraries = with pkgs.python3Packages; [ selenium ];
 } ''
 from selenium import webdriver
 from selenium.webdriver.common.by import By
 from selenium.webdriver.firefox.options import Options
 from selenium.webdriver.support.ui import WebDriverWait

 options = Options()
 options.add_argument("--headless")
 service = webdriver.FirefoxService(executable_path="${lib.getExe pkgs.geckodriver}") # noqa: E501

 driver = webdriver.Firefox(options=options, service=service)
 driver.implicitly_wait(10)
 driver.get("http://localhost:8080/web/dashboard")

 wait = WebDriverWait(driver, 60)

 # Look for some elements that should be rendered by the Javascript bundle in the UI
 assert len(driver.find_elements(By.CLASS_NAME, "mat-button-wrapper")) > 0
 assert len(driver.find_elements(By.CLASS_NAME, "top-bar")) > 0

 driver.close()
 '';
 in
 with pkgs; [ curl firefox-unwrapped geckodriver seleniumScript ];
 };
 };
 # This is the test code that will check if our service is running correctly:
 testScript = ''
 start_all()

 # Wait for InfluxDB to be available
 machine.wait_for_unit("influxdb2")
 machine.wait_for_open_port(8086)

 # Wait for Scrutiny to be available
 machine.wait_for_unit("scrutiny")
 machine.wait_for_open_port(8080)

 # Ensure the API responds as we expect
 output = machine.succeed("curl localhost:8080/api/health")
 assert output == '{"success":true}'

 # Start the collector service to send some metrics
 collect = machine.succeed("systemctl start scrutiny-collector.service")

 # Ensure the application is actually rendered by the Javascript
 machine.succeed("PYTHONUNBUFFERED=1 selenium-script")
 '';
}

This relatively short code snippet will take care of:

Building a dedicated virtual machine according to the machine spec
Starting the VM
Running the tests inside the VM (specified in `testScript)
Collecting the results

I ended up adding this check in a Github Actions workflow so that it’s run each time I make a change to my flake.

Try It! #

You can give this a go too! If you’ve got a NixOS machine, or perhaps a virtual machine you’ve been experimenting with, then you need only add the following to your flake:

1
2
3
4
5
6
7
8
9


{
 inputs = {
 nixpkgs-unstable.url = "github:nixos/nixpkgs/nixos-unstable";
 jnsgruk.url = "github:jnsgruk/nixos-config";
 jnsgruk.inputs.nixpkgs.follows = "nixpkgs-unstable";
 # ...
 }
 # ...
}

And in your NixOS machine configuration:

1
2
3
4
5
6


# Where 'inputs' is the inputs attribute of your flake
{ inputs, ... }: {
 imports = [ inputs.jnsgruk.nixosModules.scrutiny ];
 nixpkgs.overlays = [ inputs.jnsgruk.overlays.additions ];
 services.scrutiny.enable = true;
}

The next time you rebuild your system, you should be able to browse to http://localhost:8080 and see some hard drive metrics!

What’s Next? #

This was a pretty quick exercise - I’d estimate at about 3 hours in total. No doubt I will find some bugs, but already in my mind are the following future improvements:

Add some more tests that exercise more of the config options
Enable notifications configuration in the module
Submit the packages/module to the upstream or to nixpkgs

I also need to go and investigate this rather sad looking hard disk…

That’s all for now! Thanks for reading if you got this far!