Home Assistant on Jon Seager

Hot Tub Monitoring with Home Assistant and ESPHome

Tue, 12 Nov 2024 00:00:00 +0000

Introduction #

This year we put a wood-fired hot tub in our back garden. It’s a total indulgence and a real treat. I originally purchased the tub for my wife, but I must admit to having grown quite fond of it myself!

The tub itself is made from Canadian Redwood Cedar. The photo below was taken shortly after it was assembled, so there is still a little bit of leakage which stopped a few days later once the wood had expanded. The 30kW wood burner is the most effective way to heat the tub (which holds around 1700l of water), but there is also a small electric heater in line with the pump & filter which are stashed behind the tub.

There is something wonderfully simple and low-tech about the whole arrangement, but naturally I wanted to get an understanding of the energy usage, and get a reading on the temperature so that I could more accurately set it up for when we wanted to use it. Part of the reasoning for understanding the energy usage was to ensure that the pump and UV filter only fire up when the house is generating enough solar energy to cover it.

A picture paints a thousand words, so you can see what I’m talking about below:

Energy Usage Monitoring #

Starting with the easy part: monitoring the energy usage for each of the components. What you can’t see in the picture above is the following:

An Elecro Vulcan 3kw heater
An evoUV 15w UV clarifier
A Crystal Enterprises CC2513 pump
A Crystal Enterprises CC3030 filter unit

The UV filter, heater and sand filter are all essentially “in line” - when the pump is switched on, water is pulled from the bottom of the tub, through the UV clarifier, sand filter and heater, and then back into the tub (not necessarily in that order!). The heater has a thermostat in it so that it can cut in/out depending on the desired temperature as the water passes through.

In reality, the 3kw heater would take 12-15 hours to heat the tub from scratch, which is why we use the burner that usually takes around 1.5-2 hours. The electric heater is good for maintaining the temperature once the initial heating has been done.

I’ve had quite a lot of success with TP-Link Tapo P110 smart plugs. They’re very cheap, and there is already a competent Home Assistant integration for them. The only downside is that they require a proprietary app for initial setup. After having bought a few of these, I learned about Tasmota, and will probably buy smart plugs compatible with this or similar open source firmwares in the future.

My friends from the Self Hosted Podcast seem to recommend buying pre-flashed smart devices from cloudfree.shop, which I’ll certainly consider next time.

All of the above said - the TP-Link plugs work great for this use-case. I’ve got one for the pump and one for the heater. The pump is configured to run for 2 hours a day during peak daylight hours, and the heater can be toggled as and when I need it. The result, when added to Home Assistant is some neat energy monitoring and the ability to toggle things on and off easily when I need:

In an alternative view that summarises power usage for my home, there is a neat section covering the usage from individual devices (all connected to smart plugs):

As you can see - the heater is incredibly power-hungry, and as a result we hardly use it. In the two hours it ran on the day shown above, it consumed almost twice the energy that my main workstation used all day. We now constrain it’s use to a couple of hours per day in the summer months when solar generation is at its peak - otherwise we just heat the tub with the wood burner as we want.

Temperature Sensor: Initial Solution #

When the tub originally arrived, I looked around for an internet-connected pool thermometer, and was quite surprised at the lack of options. I had assumed this was something relatively common. I ended up ordering an Inkbird IBS-P02R kit.

The kit comprises a small floating thermometer, and a display unit that communicates wirelessly with the thermometer. I picked this particular model because the display unit can be connected to Wi-Fi and then checked on through an app. I had (naively…) assumed that I’d be able to get access through some sort of API, but that turned out not to be the case.

Overall, this solution worked okay. The temperature is reported in 5/10/15 minute intervals according to the configuration, and I had no issues with range/connectivity even though the tub is some distance from my house (and the house is heavily insulated with triple glazing!).

The Inkbird app for iOS leaves a lot to be desired, though. I found the user experience pretty frustrating. The web service was often slow to respond, and it required an annoying number of clicks to get to the information I wanted.

While researching the different ways I could get access to the information, I re-discovered the Inkbird IBS-P01B which is a very similar thermometer to the one I’d bought, but communicates over bluetooth. I’d initially ruled this out since my server is well out of bluetooth range, but I’d also been looking for an excuse to play with an ESP32-based microcontroller…

Temperature Sensor: Homemade Solution #

After a little bit of research I ended up ordering the following:

1x Inkbird IBS-P01B thermometer
1x ESP32-WROOM-32U development board

The former was ordered from AliExpress, and the latter from Amazon. I chose the ESP32-WROOM-32U specifically because it has an external antenna. Given the eventual placement of the ESP32 and the location of the hot tub, I wanted to maximise the chances of establishing a good bluetooth connection.

This was to be my first foray into ESP32/Arduino type development. I’d read (and heard…) lots about ESPHome, and given my goal was to integrate with Home Assistant, this felt like the right route.

Connecting the ESP32 #

The first step was to flash ESPHome onto the device. The development board I bought comes with a CP2012 USB to UART bridge on the board, so connecting the device to my machine was as simple as plugging it in with a MicroUSB cable.

Before the device can be flashed, it needs to be put into programming mode. In my case, the board has a handy BOOT button. The process for entering programming mode was therefore (starting with the device unplugged and powered down):

Press and hold the BOOT button
Connect the ESP32 over USB
Wait a few seconds
Release the BOOT button

Other development boards may not have a button, but all it really does is bridge GPIO0 and GND on the board which can be done with a wire, too. There are good docs on the connection process on the ESPHome website.

ESPHome Support #

ESPHome doesn’t natively support the IBS-P01B, but after some reading it seemed that the underlying messaging format/protocol is similar enough to the IBS-TH1 and IBS-TH2 that the same configuration can be used.

I also came across this blog post where the author had manually created an ESPHome configuration that appeared to work, and with the exact same hardware.

Nonetheless - to my mind, simpler is better most of the time, so I set about creating a simple configuration using the supported ESPHome platform.

ESPHome Dashboard #

There are a few different ways to get started with ESPHome. You can “install” using Docker, which seems to be the preferred method (passing through /dev/ttyUSBx to the container for USB connectivity), or manually using Python packages.

I tried to follow the instructions to get started with Home Assistant, but it seems this is not compatible with “unsupervised” Home Assistant servers like mine.

The esphome package is readily available in nixpkgs, so I was able to get started and fire up the dashboard like so:

❯ nix run unstable#esphome -- dashboard .
2024-11-12 14:47:19,072 INFO Starting dashboard web server on http://0.0.0.0:6052 and configuration dir ....

Browsing to http://localhost:6052 yielded me the following page:

Creating a Firmware #

I proceeded to create a new device, and entered some basic information about my Wifi network:

This is where things got interesting (for me, at least!). It turns out that the ESPHome dashboard makes use of the Web Serial API to program the ESP32 chip over USB from the browser. Given that my board was already connected and in programming mode, this was pretty simple!

The first time I tried this, I got a permission error. After looking through the messages in my kernel’s ring buffer with journalctl -k, and looking at the permissions on the /dev/ttyUSB0 device, it seemed likely that the issue was the root:root ownership with limited permissions. I solved this by running the following:

1
2


sudo chown root:users /dev/ttyUSB0
sudo chmod g+rwx /dev/ttyUSB0

This changed the group of the serial device to the same group as my user, then gave the group read, write and execute permissions. I retried in the browser and the device was flashed with the new firmware, connecting it to my network! Once the device was online, it appeared as such in the ESPHome dashboard, and I cooked up the following configuration by studing the docs on the inkbird_ibsth1_mini sensor component:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44


substitutions:
 name: esphome-web-caf3b8
 friendly_name: Tub Monitor

esphome:
 name: ${name}
 friendly_name: ${friendly_name}
 min_version: 2024.6.0
 name_add_mac_suffix: false
 project:
 name: esphome.web
 version: dev

esp32:
 board: esp32dev
 framework:
 type: arduino

web_server:
improv_serial:
logger:
api:
ota:
 - platform: esphome

wifi:
 # Pulled in from a separate secrets.yaml
 ssid: !secret wifi_ssid
 password: !secret wifi_password

esp32_ble_tracker:
 scan_parameters:
 active: true
 continuous: true
 duration: 1min

sensor:
 - platform: inkbird_ibsth1_mini
 # Bluetooth MAC address of my IBS-P01B
 mac_address: <redacted>
 temperature:
 name: "Tub Temperature"
 battery_level:
 name: "Tub Monitor Battery"

A few points on the above:

The device setup wizard created a secrets.yaml which can hold secrets to be referenced with the !secret <secret name> syntax. This is the same format as used by Home Assistant.
Specifying web_server: enables a small embedded web server that can be used to check status (see below!).
I had to specify the MAC address of the pool thermometer, which I collected using the Bluetooth Inspector app for iOS.

I clicked Install in the top right of the window, and the firmware was compiled and flashed to the device! This time I was able to flash over Wifi since the device was now connected to my network as a result of the onboarding process:

After selecting “Wirelessly”, and waiting for the firmware to be flashed, the logs shortly started flowing in:

To double check things were working, I then browsed to the IP address of the ESPHome device, and was greeted with a simple page that shows the current state of the device:

Adding to Home Assistant was as simple as adding a new integration of type ESPHome and specifying the IP address of the device!

3D Printing an Enclosure #

The final part of my build was to create a small enclosure for the device that would house the antenna correctly. I found a nice model on Printables, which printed pretty fast on my BambuLab X1 Carbon printer.

The photo below shows the device in place on the window sill, with the hot tub just visible in the background:

Home Assistant Dashboard #

With all this in place, all that remained was tying the information into my Home Assistant dashboard:

The above shows the hot tub temperature, as well as the battery level of the IBS-P01 sensor. Beneath, I’ve added basic controls and measurements available through the Tapo Home Assistant integration, which allows me to quickly toggle the pump and heater, and see their daily energy usage at a glance.

If you look closely, you’ll see the room temperatures resulting from the Roth underfloor heating integration I wrote about previously.

Summary #

We’re super pleased with our tub - it was built by Andy from Rustic Tubs. If you’re considering one and you’re in the UK, you could do a lot worse. It’s beautifully crafted, and Andy was really helpful and communicative throughout the process.

This was my first foray into ESP32-based projects, and I was pleasantly surprised with how seamless the process was. The ESPHome docs were clear, and the no-hassle flashing through the browser was a nice way to get started.

I hope this post is useful to people who are just getting started, and feel free to reach out if I’ve missed something!

Writing a Home Assistant Core Integration: Part 2

Wed, 16 Oct 2024 00:00:00 +0000

Introduction #

In my last post I described the first steps toward creating a new Home Assistant integration for the underfloor heating system in my house. In that post I outlined in detail how I set about creating a Python client for the API provided by the underfloor heating controller vendor.

In this post, I’ll describe the development setup, project structure and contribution process for building and landing a Home Assistant Core integration. I don’t consider myself an expert here, but I’ve documented my journey here in the hope that my experience might be useful to potential future contributors.

The finished integration can be seen in the Home Assistant docs under the name touchline_sl, and the code can be found on Github.

Development Setup #

The Home Assistant documentation recommends the use of Visual Studio Code with a devcontainer. This was a very quick way to get a working environment up and running, especially given that I already use Visual Studio Code for most of my programming, so I was immediately familiar.

The repository provides some tasks to help get started, including a task named Run Home Assistant Core, which takes care of setting up the runtime environment, installing dependencies and starting the server. Neat!

There are also a set of pre-commit hooks set up to ensure you don’t make any common mistakes, accidentally violate the formatting/static-typing rules for the project, forget to update requirements.txt files, etc.

This turned out to be a really nice way to get started, and if you’re new to Home Assistant Core development, I’d recommend giving this “batteries-included” approach a go. If it’s not for you, the project provides manual setup instructions too.

Integration Basics #

According to the documentation:

[An] integration is responsible for a specific domain within Home Assistant. Integrations can listen for or trigger events, offer actions, and maintain states.

Where a domain is…

a short name consisting of characters and underscores. This domain has to be unique and cannot be changed.

Err, right… while this is an accurate statement, it’s perhaps not the most enlightening for the budding new integration developer! At their core, integrations are Python modules that take information about a given system (like an underfloor heating system, or a smart plug, or a light bulb) and represent information about them in a format compatible with one of Home Assistant’s archetypes for devices/sensors/platforms/entities.

In this case, we’ll be representing Climate Entities, which have the sort of properties you might expect - target_temperature, current_humidity, current_temperature, etc.

To use Home Assistant’s terms, in my setup:

The platform is the underfloor heating system
The platform consists of some devices, in this case the physical thermostats in each room of my house
The devices each represent one or more climate entities (humidity, temperature, etc.)

File Structure #

The basic file structure can be laid down with some scaffold tooling, but even in its finished state, my integration doesn’t have many files:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


.
├── __init__.py # the component file
├── climate.py # ties info from the api into home assistant terms
├── config_flow.py # defines the fields/flow for integration config
├── const.py # constants used across the integration
├── coordinator.py # data update coordinator
├── manifest.json # defines project dependencies and metadata
├── strings.json # defines strings displayed in various ui elements
└── translations # a directory containing one file per language
 └── en.json # english translation of strings.json

And to give an idea of the scale of the project in its completed form:

----------------------------------------------------------------------
Language files blank comment code
----------------------------------------------------------------------
Python 5 71 27 215
JSON 3 0 0 82
----------------------------------------------------------------------
SUM: 8 71 27 297
----------------------------------------------------------------------

`manifest.json` #

Starting with the most simple first! The manifest.json describes the integration: what its name is, where its documentation is found, who owns the code and the libraries it depends on:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


{
 "domain": "touchline_sl",
 "name": "Roth Touchline SL",
 "codeowners": ["@jnsgruk"],
 "config_flow": true,
 "documentation": "https://www.home-assistant.io/integrations/touchline_sl",
 "integration_type": "hub",
 "iot_class": "cloud_polling",
 "requirements": ["pytouchlinesl==0.1.8"]
}

Here I selected cloud_polling as the iot_class, because my integration reaches out periodically to the API, polling for new information. You’ll note also that the integration requires my pytouchlinesl library in order to run.

`init.py` #

Typically for integrations, this file defines how to setup the integration, and how to unload it. You can see the full source of my implementation on Github.

I define an async_setup_entry() method which takes care of:

Establishing a coordinator per TouchlineSL module
Performing the initial hydration of data
Registering each TouchlineSL module as a device in Home Assistant’s Device Registry, which is where Home Assistant “keeps track of devices”

There is a shortened, annotated version of the method below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


async def async_setup_entry(hass: HomeAssistant, entry: TouchlineSLConfigEntry) -> bool:
 """Set up Roth Touchline SL from a config entry."""
 account = TouchlineSL(...)

 coordinators: list[TouchlineSLModuleCoordinator] = [
 TouchlineSLModuleCoordinator(hass, module)
 for module in await account.modules()
 ]
 # ...
 device_registry = dr.async_get(hass)
 # Create a new Device for each coorodinator to represent each module
 for c in coordinators:
 module = c.data.module
 device_registry.async_get_or_create(
 config_entry_id=entry.entry_id,
 identifiers={(DOMAIN, module.id)},
 name=module.name,
 # ...
 )

 entry.runtime_data = coordinators
 await hass.config_entries.async_forward_entry_setups(entry, PLATFORMS)
 return True

Before returning async_setup_entry() invokes async_forward_entry_setups(). This ensures that async_setup_entry() in climate.py is called to ensure each of the climate entities is registered.

The PLATFORMS variable is a list of platform types that the integration supports, in this case a single-item list containing just Platform.CLIMATE, which is how async_forward_entry_setups knows to invoke the async_setup_entry() method in climate.py:

1
2
3
4


from homeassistant.const import Platform
# ...
PLATFORMS: list[Platform] = [Platform.CLIMATE]
# ...

To initialise the integration, the user must authenticate with the Roth API so that module details are fetched before constructing the coordinator. Thus, before this code is executed, the user must go through the config flow…

`config_flow.py` #

Despite the final implementation being quite simple, this is probably one of the areas I found most challenging to get right. There are some docs but they only scratch the surface, and implementations in other integrations seem to vary quite dramatically (mostly depending on when they were written).

I went through a few iterations of this config flow, mostly because I had originally implemented the ability to select a specific module from the user’s account. The review process guided me toward simply logging into the account, then enrolling each of the modules associated with the account - since users can always disable entities they don’t wish to manage in Home Assistant.

The config_flow.py defines which input fields need to be presented to the user, and then passes on the relevant information needed to set up the integration. In my implementation, the code:

Prompts the user for their username and password
Authenticates with the service and fetches the user’s unique ID
Registers that unique ID, aborting if the specified account has already been used
Creates a config entry in Home Assistant that stores the user’s credentials

The config flow also has some basic error handling that can distinguish the difference between poor credentials, networking issues, etc. The full implementation can be seen on Github, but the important parts are highlighted in the following snippet:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


# ...
class TouchlineSLConfigFlow(ConfigFlow, domain=DOMAIN):
 # ...
 async def async_step_user(
 self, user_input: dict[str, Any] | None = None
 ) -> ConfigFlowResult:
 # ...
 if user_input is not None:
 try:
 account = TouchlineSL(
 username=user_input[CONF_USERNAME],
 password=user_input[CONF_PASSWORD],
 )
 # Use the credentials to fetch unique user id
 await account.user_id()
 except RothAPIError as e:
 # ...
 else:
 # Set unique ID, abort setup if already used
 unique_account_id = await account.user_id()
 await self.async_set_unique_id(str(unique_account_id))
 self._abort_if_unique_id_configured()

 # Create a config entry containing the user's credentials
 return self.async_create_entry(
 title=user_input[CONF_USERNAME], data=user_input
 )

 return self.async_show_form(
 step_id="user", data_schema=STEP_USER_DATA_SCHEMA, errors=errors
 )

What caught me out was how the fields are given titles/descriptions. These attributes are all configured in the strings.json, where the config map contains keys for each of the config “steps”.

The code above defines a step named user, since the method name is async_step_user. The step’s name, description and input fields are defined in the strings.json:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


{
 "config": {
 "flow_title": "Touchline SL Setup Flow",
 "error": {
 "cannot_connect": "[%key:common::config_flow::error::cannot_connect%]"
 // ...
 },
 "step": {
 "user": {
 "title": "Login to Touchline SL",
 "description": "Your credentials for the Roth Touchline SL mobile app/web service",
 "data": {
 "username": "[%key:common::config_flow::data::username%]",
 "password": "[%key:common::config_flow::data::password%]"
 }
 }
 },
 "abort": {
 "already_configured": "[%key:common::config_flow::abort::already_configured_device%]"
 }
 }
 // ...
}

The values displayed to the user are pulled from the translation files at runtime depending on their language configuration. In my integration, the corresponding translations/en.json contains the following fields that map to those defined above:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


{
 "config": {
 "flow_title": "Touchline SL Setup Flow",
 "error": {
 "cannot_connect": "Failed to connect"
 // ...
 },
 "step": {
 "user": {
 "data": {
 "password": "Password",
 "username": "Username"
 },
 "description": "Your credentials for the Roth Touchline SL mobile app/web service",
 "title": "Login to Touchline SL"
 }
 },
 "abort": {
 "already_configured": "Device is already configured"
 }
 }
 // ...
}

In hindsight, this is covered in the docs, but it definitely didn’t click with me when I was going through it, so it seemed worth calling out! The net result of this setup is a configuration dialog that looks like so:

`coordinator.py` #

This was an addition I made during the review process (more on that later), and appears to be the preferred way to implement the fetching of data from upstream APIs. By implementing a DataUpdateCoordinator, Home Assistant can ensure that a single coordinated poll happens across all entities managed by an integration. If an integration manages many entities for which it needs to fetch/update details, the coordinator helps ensure that the API is called only as often as is needed.

The coordinator class is very simple: it defines a single method _async_update_data which returns the data for the device its coordinating. As the developer, you can specify the type of the data returned by the coordinator. I chose to represent this as a Python dataclass:

1
2
3
4
5


@dataclass
class TouchlineSLModuleData:
 module: Module
 zones: dict[int, Zone]
 schedules: dict[str, GlobalScheduleModel]

The coordinator is initialised with some basic information such as a name and an update interval:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# ...
class TouchlineSLModuleCoordinator(DataUpdateCoordinator[TouchlineSLModuleData]):
 def __init__(self, hass: HomeAssistant, module: Module) -> None:
 super().__init__(
 hass,
 logger=_LOGGER,
 name=f"Touchline SL ({module.name})",
 update_interval=timedelta(seconds=30),
 )

 self.module = module

The _async_update_data method then queries the API, and returns data in the newly defined format:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


async def _async_update_data(self) -> TouchlineSLModuleData:
 try:
 zones = await self.module.zones()
 schedules = await self.module.schedules()
 except RothAPIError as error:
 # ...
 # Return the data using our dataclass from above
 return TouchlineSLModuleData(
 module=self.module,
 zones={z.id: z for z in zones},
 schedules={s.name: s for s in schedules},
 )

You can see the full implementation on Github.

`climate.py` #

And finally, on to the business logic of tying fields from the upstream API into the relevant attributes in Home Assistant!

The first task handled by this file is registering each of the climate entities by iterating over each zone, in each coordinator’s module:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# ...
async def async_setup_entry(
 hass: HomeAssistant,
 entry: TouchlineSLConfigEntry,
 async_add_entities: AddEntitiesCallback,
) -> None:
 """Set up the Touchline devices."""
 coordinators = entry.runtime_data
 async_add_entities(
 TouchlineSLZone(coordinator=coordinator, zone_id=zone_id)
 for coordinator in coordinators
 for zone_id in coordinator.data.zones
 )
# ...

Home Assistant entities have well-defined APIs - the docs for climate entities show the supported attributes and their data types. I used a combination of the docs, and the source code to establish how to implement my ClimateEntity, which boiled down to the following interface:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


class TouchlineSLZone(CoordinatorEntity[TouchlineSLModuleCoordinator], ClimateEntity):
 # Construct a Touchline SL climate zone.
 def __init__(self, coordinator: TouchlineSLModuleCoordinator, zone_id: int) -> None:
 # Handle updated data from the coordinator.
 @callback
 def _handle_coordinator_update(self) -> None:
 # Return the device object from the coordinator data.
 @property
 def zone(self) -> Zone
 # Report if the device is available.
 @property
 def available(self) -> bool
 #Set new target temperature.
 async def async_set_temperature(self, **kwargs: Any) -> None
 # Assign the zone to a particular global schedule.
 async def async_set_preset_mode(self, preset_mode: str) -> None
 # Populate attributes with data from the coordinator.
 def set_attr(self) -> None:

Arguably the most important part here is set_attr(). which takes care of mapping fields from the objects provided by my pytouchlinesl library to attributes on the climate entity:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


def set_attr(self) -> None:
 """Populate attributes with data from the coordinator."""
 schedule_names = self.coordinator.data.schedules.keys()

 self._attr_current_temperature = self.zone.temperature
 self._attr_target_temperature = self.zone.target_temperature
 self._attr_current_humidity = int(self.zone.humidity)
 self._attr_preset_modes = [*schedule_names, CONSTANT_TEMPERATURE]

 if self.zone.mode == "constantTemp":
 self._attr_preset_mode = CONSTANT_TEMPERATURE
 elif self.zone.mode == "globalSchedule":
 schedule = self.zone.schedule
 self._attr_preset_mode = schedule.name

The full implementation is on Github.

Testing #

Testing is (understandably) a little complicated. The requirements for landing code for an integration stipulate that you must include unit tests for any config/options flows. In my case, this meant the following:

Implementing a mock TouchlineSL client fixture to ensure that the unit tests don’t try to reach out to the real Roth API
Implementing a mock config entry fixture
A unit test for a successful config flow execution
A parametrised unit test for unsuccessful config flow due to possible exceptions when hitting the API
A unit test to ensure that multiple config flows resulting in the same user ID fail

To me this feels like the bare minimum, and unfortunately doesn’t really provide any confidence that the integration actually functions correctly. I’m hoping to improve this in the future, but for now further testing has been manual.

Docs & Brand #

As I was creating the Pull Request to contribute my integration, I was prompted by the template to link to further PRs that added the documentation and brand assets for my integration. I used the existing touchline docs as a template and modified them for my integration. The docs pull request added a single file touchline_sl.markdown containing 29 lines, which results in some nicely rendered docs on the Home Assistant website:

There were already branding assets for Roth as part of the touchlinesl integration that was previously merged. Based on some review feedback, I created a new Integration Brand named roth, with which both the touchline and touchline_sl integrations are associated. This has the nice effect of grouping them when setting up a new integration:

Contribution Process #

I submitted my first efforts for review in home-assistant/core#124557. The checklist guided me nicely through what needed to be done, and overall the process went pretty smoothly, and pretty quickly (despite the 117 comments!). The process took a little under two days in total. I was also fortunate with my timing, since my code landed the day before the next beta release was cut, so it shipped relatively quickly.

I’d like to offer my sincere thanks to @joostlek who not only reviewed my code extremely quickly, but took the time to explain things to me both in the Github PR, but also by proactively reaching out to me on the Home Assistant Discord, which I really appreciated.

I got a lot of feedback, which I expected. This was my first attempt at writing code for Home Assistant, and it’s a pretty large and well established project. I do think the project could do with some better developer documentation, which would dramatically reduce the burden of effort on reviewers and give contributors a better chance of “getting it right”.

One might argue I could have done more reading and more research - though I spent a good amount of time reading both the documentation and the source code of other integrations. I’m not sure I’d ever have reached the conclusions that @joostlek kindly nudged me toward.

Overall, my implementation was more brief, more simple and more efficient as a result of the review process, and based on my experience I’d advocate for having a go if you’ve been on the fence! Often submitting code to such a project can be daunting, but as with my experience when contributing to nixpkgs, if you go in with an open mind you’re sure to learn something from the process.

Results #

I’m really pleased with the results. However obtuse the developer experience felt at times, there is no denying that what I was able to get from my 297 lines of implementation code is quite staggering. I was super impressed that just by following the conventions I was able to get such intuitive controls, a full graphable history of temperatures and such a wide variety of ways to display the information in my various dashboards.

Once set up, you’re able to get a view of all the different zones imported by the module (the names of each zone are pulled from the upstream API if the zones are named, and each can be associated with a given area in Home Assistant):

The default dashboard displays thermostat controls for each zone. These allow you to see the current and target temprature, as well as adjust the target temperature if you need to:

Expanding the thermostat cards gives you a more detailed view, showing the mode and the “Preset”. My implementation maps “Presets” to “Global Schedules” configured in the Roth module:

Summary #

And that’s a wrap! I learned a bunch writing this integration, and the resulting user experience is quite a lot better than I get with the default Roth application.

My next mission is to use the information from these climate entities to automate opening the Velux windows in the roof when things get warm in the summer, but I’ve got a lot to learn about Home Assistant in the mean time!

Writing a Home Assistant Core Integration: Part 1

Wed, 11 Sep 2024 00:00:00 +0000

Introduction #

Back in March, my family and I moved into a new home. It’s a modern construction which came with solar panels (and associated inverter/battery storage), and uses an air source heat pump to heat the house with underfloor heating. Being a new renovation, nearly all of the appliances and components in the house have a form of internet connectivity (some more useful than others!).

Since day 1, I’ve been hoping to consolidate all of the various applications, data feeds and functions into one single place. I’ve been a long-time listener to the Self Hosted podcast, which often extols the virtues of Home Assistant. I’ve got no prior experience with Home Assistant, but for the last three months I’ve been running it on my home server, with a collection of custom integrations and hacks that enable me to control the underfloor heating and solar inverter.

The underfloor heating controller is a Roth Touchline SL system. In my set up, there is a single “module” which represents my house, and a number of “zones” which represent different rooms.

There was unfortunately no integration for this system in Home Assistant - there is one for the previous generation “Roth Touchline”, but this appears to function over the LAN, whereas the Touchline SL system is controlled over the internet using their API.

After reading the source code of a few other climate integrations in Home Assistant, it became clear to me that the first step was to create a Python client for the API which could be used in the integration.

This post will cover the design, implementation and limitations of the library I wrote: pytouchlinesl. If you came here to read about writing code for Home Assistant, you’ll have to wait for the next post! 😉

Designing the library #

Upstream API #

Usually, one would interact with a Roth Touchline SL system through their mobile apps, or through their online portal. The mobile app seems to be quite a lightweight wrapper around the web application, and I’ve not been able to detect any difference in functionality.

A bit of searching uncovered that Roth also provide an API for the Touchline SL system, and an OpenAPI spec. This made the process significantly easier, though there are some discrepancies in what is documented compared with how the API actually behaves. It feels to me like the API may have evolved, and the documentation has remained static - or perhaps it was always inaccurate? Either way, I spent quite a bit of time manually fuzzing the API to work out the correct set of parameters for some endpoints.

I also studied the web application using the Chrome Dev Tools. Of all the endpoints documented, it seemed like I’d only need the following:

POST /authentication: authenticates with the API, taking a username and password, and returning a token
GET /users/{user_id}/modules: returns a list of modules associated with the user
GET /users/{user_id}/modules/{module_udid}: returns details of a specific module (zones, schedules, etc.)

There is a slight awkwardness here, in that the last endpoint returns all of the data - for all zones, all schedules, etc. This feels inefficient, but I couldn’t find a way of getting information about a specific zone, or a specific schedule. The web application seems to rely upon polling the (undocumented) endpoint GET /users/{user_id}/modules/{module_id}/update/data/parents/[]/alarm_ids/[]/last_update/{timestamp}, which delivers deltas in the data since a given timestamp. This is useful in the context of the app because it can request the full dataset once, then request only changes from that point onwards, keeping the app state up to date without requesting the whole dataset.

Making changes to the configuration of zones and their temperatures is also fragmented. In essence, one can:

Set a zone to a constant temperature: POST /users/{user_id}/modules/{module_udid}/zones
Place a zone on a global schedule: POST /users/{user_id}/modules/{module_udid}/zones/{zone_id}/global_schedule
Place a zone on a local schedule: POST /users/{user_id}/modules/{module_udid}/zones/{zone_id}/local_schedule

The first is self-explanatory, enabling a zone to be set to a constant temperature (19.0C, for example). Touchline SL modules also support “schedules” which contain time periods for the specified zones to reach certain temperatures. In the case of a “Global Schedule”, multiple zones can be assigned, while a “Local Schedule” is specific to a single zone. The awkwardness in the API here is that to “add” a zone to a global schedule, you must re-specify the entire schedule, and specify all of the zones that should be on the schedule…

Basic Requirements #

In order to fulfil the basic functionality of my (future) Home Assistant integration, I limited the requirements of the first version to:

Authenticate with the API using a username and password
List modules associated with the account
Get a specific module
Get a specific zone
Get a list of global schedules
Get a specific global schedule
Assign a constant temperature to a zone
Assign a zone to a specific global schedule

I don’t use local schedules in my system, so I’ve omitted them for now, though updating the library to support them would be trivial.

Outline design/experience #

With those requirements in mind, I came up with a rough sketch of how I’d like the library to behave:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


tsl = TouchlineSL(username="foo", password="bar")
module = await tsl.module(id="deadbeef")

lounge = module.zone_by_name("Lounge")
kitchen = module.zone(1234)

# Properties should be available such as:
# lounge.current_temperature
# kitchen.humidity

await kitchen.set_temperature(temperature=20.0)

living_spaces = await tsl.schedule_by_name(schedule_name="Living Spaces")
await lounge.set_schedule(living_spaces.id)

And from that came a reasonable outline of the API for the library:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


class TouchlineSL:
 # Construct a class that represents a Touchline SL account
 def __init__( self, *, username: str | None = None, password: str | None = None)
 # Get a list of modules associated with the account
 async def modules(self) -> list[Module]
 # Get a specific module, by ID
 async def module(self, *, module_id: str) -> Module | None

class Module:
 # Get a list of zones from the module, optionally including disabled zones
 async def zones(self, *, include_off: bool = False) -> list[Zone]
 # Get a specific zone, by ID
 async def zone(self, zone_id: int) -> Zone | None
 # Get a specific zone, by name
 async def zone_by_name(self, zone_name: str) -> Zone | None
 # Get a list of global schedules
 async def schedules(self) -> list[Schedule]:
 # Get a specific schedule, by ID
 async def schedule(self, schedule_id: int) -> Schedule | None
 # Get a specific schedule, by name
 async def schedule_by_name(self, schedule_name: int) -> Schedule | None

class Zone:
 # Get the schedule the zone is assigned to
 def schedule(self) -> Schedule | None
 # Set the zone to a constant temperature
 async def set_temperature(self, temperature: float)
 # Assign the zone to a specific schedule
 async def set_schedule(self, schedule_id: int)

Note that after reading the code from other climate integrations in Home Assistant, it became clear to me that they favour the use of async libraries, and thus my library was designed to use asyncio from the start.

Python tools/libraries #

There are a couple of things I’ve found tiresome about Python over the years, but things do seem to be looking up. I’ve always found the package management and distribution to be awkward, and I can’t be the only one if the number of projects looking to target that problem is anything to go by (e.g. poetry, rye, uv, pdm, etc.).

Part of this seems to come from fractures in the community itself - there (still!) appears to be disagreements surrounding PEPs such as PEP-621 which introduced pyproject.toml as a way of managing project metadata and dependencies, with the maintainers of some high-profile and widely adopted libraries refusing to adopt it.

That said, there are a couple of things I’ve been meaning to try in anger, and this project was a good opportunity to do so:

`uv` #

Developed by Astral, uv is the “new shiny” at the time of writing, and I can understand why. Pitched as “Cargo, but for Python”, it aims to solve a myriad of problems in the Python ecosystem. uv can handle the download/install of multiple Python versions, the creation of virtual environments, running Python tools in a one-off fashion (like pipx), locking dependencies deeply in a project (by hash) and still maintains a pip compatible command-line experience with uv pip. To add to all of that, it’s ridiculously fast; on a couple of occasions I’ve actually found myself wondering if it did anything when installing dependencies for large projects, because it’s so much faster than I’m used to.

`pydantic` #

Pydantic is a data validation library for Python. It’s entirely driven by Python’s type-hints which means that you get nice integration with language servers. Pydantic allows you define data models in native Python, but emit standard JSON Schema docs for models. It’s integrated quite widely across the Python ecosystem, and to me feels like it bridges the gap between what I hoped type annotations would do for Python, and what they actually do in reality!

`ruff` #

I’ve been using this one for at least the last year for all my Python linting and formatting needs, but I still feel it deserves a call out. There have been a couple of small changes to command line API and such along the way, but overall I’ve found it to be a dramatic improvement over my last setup - which comprised of black, isort, flake8 and a pile of plugins. I had no particular beef with black, but I find flake8’s lack of pyproject.toml support irritating, and grew tired of plugins failing as flake8 released new versions.

In my experience, ruff is stupid fast, and because it ships flake8-compatible rules for all of the plugins I was using as one bundle, they never break. It’s also nice to just have one tool to use everywhere. If you’re interested, you can see how I configure ruff in the pyproject.toml.

Implementation #

With a basic design in mind, and tooling ready to go, I set about building the library itself. It was now time to reconcile my intended design with the realities of the provisions made by the upstream API.

I mentioned previously that the only useful endpoint for getting information about zones/schedules would in fact return all of the data for a given module. Too many calls to this endpoint would likely result in poor performance, so I wanted to introduce some basic caching along the way.

Client implementation #

For the underlying API client implementation, I opted for the following:

BaseClient: a class which inherits from Python’s abc.ABC. This enables the creation of multiple client implementations by defining of the set of methods/properties that any client interacting with the Roth API should define. This decision is primarily to support testing through dependency injection, rather than mocking with patches (more details on that later).
RothAPI: a concrete implementation of the BaseClient abstract class. It is here that I built the actual implementation of the client which handles authentication, GETing and POSTing data, caching, and marshalling API responses into the correct types (defined with Pydantic).

Also included in the client package is the models package. The models package contains (mostly) auto-generated Pydantic models based on real-life responses I got from the API. This is really a function of laziness, but was a convenient way to get type annotated models for the responses I was receiving from the API. Each time I hit a new endpoint, I took the JSON result and did a quick conversion with https://jsontopydantic.com/, before manually adjusting names and updating some fields with Literals.

Caching #

I mentioned earlier that I wanted to implement some basic caching. While I am aware of various plugins for aiohttp (and other request libraries) that could handle this for me, my requirements were quite simple, so I chose to just build it into the library. In this case, caching is implemented on the Module class. This is because the large blob of data that is requested to populate details about a module and its zones/schedules is requested per module.

The caching works like this:

The Module class has “private” attributes named _raw_data and _last_fetched:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


class Module:
 def __init__(
 self,
 *,
 client: BaseClient,
 module_data: AccountModuleModel,
 cache_validity: int = 30,
 ):
 # ...

 # Raw data about the zones, schedules, tiles in the module
 self._raw_data: ModuleModel
 # Unix timestamp representing the last time the _raw_data was fetched
 self._last_fetched = 0
 self._cache_validity = cache_validity

 # ...

There is only one method on this class that calls the underlying client, and that’s another “private” method named _data. This method takes an optional refresh keyword argument, which forces the _raw_data attribute to be updated, but by default will only fetch data if the cached data has expired (after the number of seconds specified in self._cache_validity). If refresh is false, and the cache hasn’t expired, it simply returns the stored raw data:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


async def _data(self, *, refresh: bool = False) -> ModuleModel:
 """Get the raw representation of the module from the upstream API.

 If the data has never been fetched from upstream, or the data is older
 than the cache validity period, then the data is refreshed using the
 upstream API.

 Args:
 refresh: (Optional): Force the data to be refreshed using the API.
 """
 if refresh or (round(time.time() * 1000) - self._last_fetched) > self._cache_validity:
 self._raw_data = await self._client.module(self.id)
 self._last_fetched = round(time.time() * 1000)

 return self._raw_data

Each of the public methods (zones(), zone(), schedule(), etc.) access the raw data through the _data() method, and pass through the refresh flag which is exposed. This means that any developer consuming this library can chose to live with the caching, or override it and force a refresh like so:

1
2
3
4
5
6
7


tsl = TouchlineSL(username="foo", password="bar")
module = await tsl.module(id="deadbeef")

# Request a zone, accepting cached data (default)
zone = await module.zone_by_name("kitchen")
# Or force the data to be refreshed using the upstream API
zone = await module.zone_by_name("kitchen", refresh=True)

Testing, CI & Publishing #

Testing #

In the previous section, I described how I’d set up an abstract base class for the API client, then created an implementation of that for the actual Roth API. One of the main reasons I like this pattern for API clients is that it simplifies testing and reduces the need for monkey patching or traditional mocking (yes, I know the fake API client is sort of a mock…).

Because TouchlineSL can optionally be constructed with any client that implements BaseClient’s abstract base class, it’s trivial to implement a fake API backend that returns fixture data to be used in testing. In this case, the fixtures are stored as JSON files in the repository, and contain real life responses received from the API.

The FakeRothAPI returns the sample data for each of the methods defined in the abstract class. The following is a partial extract from the fake client code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


# ...
from pytouchlinesl.client import BaseClient
from pytouchlinesl.client.models import AccountModuleModel, ModuleModel

data_dir = Path(os.path.realpath(__file__)).parent / "sample-data"


class FakeRothAPI(BaseClient):
 def __init__(self):
 self._user_id = 123456789
 self._token = "deadbeef"

 async def user_id(self) -> int:
 return self._user_id

 async def authenticated(self) -> bool:
 return True if self._token else False

 async def modules(self) -> list[AccountModuleModel]:
 with open(data_dir / "modules.json", "r") as f:
 data = json.load(f)

 return [AccountModuleModel(**m) for m in data]

#...

From there, I defined a number of test fixtures using pytest’s @pytest.fixture decorator, which provide an initialised TouchlineSL instance, backed by the fake client, a Module instance, and a Zone instance.

From there, the tests are fairly simple. Beyond injecting the fake client, there is no mocking required, which in my opinion keeps the test code much easier to read and understand. It also means I could focus more of my energy on validating the logic I was testing, rather than worrying about how patching might interact with the rest of the code. I’ve felt bad about mocking for years, and I think the clearest articulation I’ve seen is “Don’t Mock What You Don’t Own” in 5 Minutes.

One aspect of the test suite I don’t love is the use of time.sleep in certain tests. This is because my caching implementation relies on reading a timestamp to decide on whether it should refresh data. In general I steer away from sleeps in tests, as they’re often used to mask an underlying non-determinism, but in this case it felt a reasonable trade-off, given that I’m testing a time-based functionality.

CI #

I wanted to ensure that any pull requests were tested, and that they conform to the project’s formatting/linting rules. Since the project is hosted on Github, I used Github Actions for this. The pipeline for this project is pretty simple, it lints and formats the code with ruff failing if any files were changed by the formatter or any of the linting rules were violated.

Finally, uv is used to run pytest across a matrix of supported Python versions. I could have used uv to handle the download and install of different Python versions too, but the setup-python actions has served me perfectly well in the past.

Publishing #

Publishing is taken care of in CI. I don’t like having to remember the magic incantation for building, authenticating and publishing locally. I want the process to be as consistent and as transparent as possible for the people consuming the project.

What was new to me this time was publishing to PyPI with a “Trusted Publisher” setup. To quote their docs:

“Trusted publishing” is our term for using the OpenID Connect (OIDC) standard to exchange short-lived identity tokens between a trusted third-party service and PyPI. This method can be used in automated environments and eliminates the need to use manually generated API tokens to authenticate with PyPI when publishing.

Rather than setting up Github Actions to hold an API token in a Secret, there is some automation that links a Github project to a PyPI project. The project doesn’t even have to be previously published on PyPI to get started, and new projects can be configured as “Pending”, then published to for the first time from your CI system of choice 🚀.

I configured Github Actions to trigger the release of pytouchlinesl on new tags being pushed:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


# ...
publish:
 name: Build and Publish to PyPI
 runs-on: ubuntu-latest
 needs:
 - tests

 # Trusted publisher setup for PyPI
 environment:
 name: pypi
 url: https://pypi.org/p/pytouchlinesl
 permissions:
 id-token: write

 steps:
 - name: Checkout the code
 uses: actions/checkout@v4

 - name: Install `uv`
 run: |
 curl -LsSf https://astral.sh/uv/install.sh | sh

 - name: Build the package
 run: |
 uvx --from build pyproject-build --installer uv

 - name: Publish to PyPi
 uses: pypa/gh-action-pypi-publish@v1.10.0

Contributing to `nixpkgs` #

Finally, my Home Assistant server runs NixOS, so in order for the integration to be packaged easily in the future, I created a small PR to include pytouchlinesl in nixpkgs. The original PR went through pretty quickly - thanks to @drupol for the fast review!

Since then I’ve made a couple of minor version bumps to the library as I discovered small issues when building the integration, but the final derivation is quite compact (see below). The Python build tooling in Nix is quite mature at this point, and I gained a fair bit of experience using it when packaging snapcraft and charmcraft.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32


#...

buildPythonPackage rec {
 pname = "pytouchlinesl";
 version = "0.1.5";
 pyproject = true;

 disabled = pythonOlder "3.10";

 src = fetchFromGitHub {
 owner = "jnsgruk";
 repo = "pytouchlinesl";
 rev = "refs/tags/${version}";
 hash = "sha256-kdLMuxA1Ig85mH7s9rlmVjEsItXxRlDA1JTFasnJogg=";
 };

 build-system = [ setuptools ];

 dependencies = [
 aiohttp
 pydantic
 ];

 nativeCheckInputs = [
 pytestCheckHook
 pytest-asyncio
 ];

 pythonImportsCheck = [ "pytouchlinesl" ];

 # ...
}

Summary #

And that concludes the first part of this series! Hopefully you found this useful - I’ve learned a lot from people over the years by understanding how they approach problems, so I thought I’d post my methodology here in case it helps anyone refine their process.

I’m certainly no Python expert, so if you’ve spotted a mistake or you think I’m wrong, get in touch.

In the next post, I’ll cover writing the Home Assistant integration, and contributing it to Home Assistant.

Home Assistant on Jon Seager

Hot Tub Monitoring with Home Assistant and ESPHome

Introduction #

Energy Usage Monitoring #

Temperature Sensor: Initial Solution #

Temperature Sensor: Homemade Solution #

Connecting the ESP32 #

ESPHome Support #

ESPHome Dashboard #

Creating a Firmware #

3D Printing an Enclosure #

Home Assistant Dashboard #

Summary #

Writing a Home Assistant Core Integration: Part 2

Introduction #

Development Setup #

Integration Basics #

File Structure #

manifest.json #

__init__.py #

config_flow.py #

coordinator.py #

climate.py #

Testing #

Docs & Brand #

Contribution Process #

Results #

Summary #

Writing a Home Assistant Core Integration: Part 1

Introduction #

Designing the library #

Upstream API #

Basic Requirements #

Outline design/experience #

Python tools/libraries #

uv #

pydantic #

ruff #

Implementation #

Client implementation #

Caching #

Testing, CI & Publishing #

Testing #

CI #

Publishing #

Contributing to nixpkgs #

Summary #

`manifest.json` #

`init.py` #

`config_flow.py` #

`coordinator.py` #

`climate.py` #

`uv` #

`pydantic` #

`ruff` #

Contributing to `nixpkgs` #