Patching and migrating systems

Overview

Documentation on how to patch and replace or migrate systems in the sudo.is infrastructure.

Systems

Home Assistant

Data paths

  • /srv/hass: The home directory of the hass user for Home Assistant.
    • /srv/hass/appdaemon: AppDaemon programs and config.
    • /srv/hass/home-assistant: Home Assistant Core.
    • /srv/hass/mqtt: Config for the reluctant MQTT broker.
    • /srv/zigbee2mqtt: Data and config for Zigbee2MQTT.
    • /srv/zwavejs: Data and config for Z-Wave JS.
  • /srv/owntone: Database for OwnTone (contains credentials for AirPlay).
  • /srv/mariadb_dumps: Backups of the MariaDB databases.
  • /srv/homeassistant_backups: Backups /srv/hass/homeassistant created by the Backup Integration.
  • /srv/zwavejs_backups: Backups of /srv/zwavejs/data created by Z-Wave JS.

First start by standing up the new host and with inventory:

oldhost:
  hass_container_state: stopped
  owntone_state: stopped

Then converge the new host:

$ ansible-playbook common.yml --limit "hass:&${newhost}" --diff
$ ansible-playbook hass.yml --limit "${newhost}" --diff

Run a backup of the MariaDB database, and rync /srv (includes both the .sql file and the HA home directory):

$ ansible $oldhost -m command -a "/usr/local/bin/mariadb_backup.sh"
$ ansible-playbook rsync-paths.yml --extra-vars '{
    "srchost": "${oldhost}",
    "dsthost": "${newhost}",
    "paths": "/srv/"
  }'

As a sanity check, validate that the dir is about the same size:

$ ansible hass -m command -a "du -sh /srv/"

This gets most of the data in place on the new host. A new MariaDB backup needs to be taken after shutting down Home Assistant on the old host, which is then imported on the new host.

Cutover steps

Downtime

Patching this system requires a brief peroid of downtime, as it is not possible to run Home Assistant in any sort of high availability configuration.

This is largely due to HA using the configuration directory to store data in .json files, separate from the data that is stored in the database.

HA needs to be shut down on the old host before a final rsync is done and, if needed, the MariaDB database is imported on the new host.

First check that $newhost and $oldhost are set:

$ echo "oldhost: ${oldhost}\nnewhost: ${newhost}"

The start the cutover:

  1. Shut down services and containers on ${oldhost}.

    $ ansible-playbook hass.yml --limit ${oldhost} --tags docker-containers,systemd-services

  2. Create a new MariaDB dump on ${oldhost}.

    $ ansible $oldhost -m command -a "/usr/local/bin/mariadb_backup.sh"

  3. While HA is down, do a final rsync of /srv:

    $ ansible-playbook rsync-paths.yml --extra-vars '{"srchost": "${oldhost}", "dsthost": "${newhost}", "paths": "/srv/"}'

  4. Import the MariaDB dump on ${newhost}:

    $ zcat /srv/mariadb_dumps/hass.sql.gz  | mariadb hass

  5. Update DNS (if needed)

  6. Start services and containers on ${newhost}:

    $ ansible-playbook hass.yml --tags docker-containers,systemd-services

After cutting over, converge the new host again:

$ ansible-playbook hass.yml --diff
$ ansible-playbook common.yml --limit hass --diff

And then monitor for issues.