Failed configuring DHCP on rack controller

[Anton Troyanov]

Running MAAS 3.5.0-beta1 in a vanilla LXD container with 2 interfaces.

Steps to reproduce:

1) create a container with 1 interface, start it and install maas.
2) stop the container and add another interface.
3) start the container and configure the interface and restart maas
4) BUM

eth1 is controlled by MAAS, and as soon as I enable DHCP via UI DHCP service goes into a crash-loop.

root@maas-deb:~# cat /etc/netplan/50-cloud-init.yaml
# This file is generated from information provided by the datasource. Changes
# to it will not persist across an instance reboot. To disable cloud-init's
# network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    version: 2
    ethernets:
        eth0:
            dhcp4: true
        eth1:
            dhcp4: false
            addresses: [10.10.10.10/24]

root@maas-deb:~# apt-cache madison maas
      maas | 1:3.5.0~beta1-16214-g.1eeea8524-0ubuntu1~22.04.1 | https://ppa.launchpadcontent.net/maas/3.5-next/ubuntu jammy/main amd64 Packages

```
Mar 05 20:47:43 maas-deb regiond[611]: maasserver.dhcp: [critical] Error configuring DHCPv4 on rack controller 'maas-deb (xgbk8d)': DHCPv4 server failed to restart: Service 'maas-dhcpd' failed to restart. Its current state is 'SERVICE_STATE.OFF' and 'dead'.
Mar 05 20:47:43 maas-deb regiond[611]: Traceback (most recent call last):
Mar 05 20:47:43 maas-deb regiond[611]: --- <exception caught here> ---
Mar 05 20:47:43 maas-deb regiond[611]: File "/usr/lib/python3/dist-packages/maasserver/dhcp.py", line 878, in configure_dhcp
Mar 05 20:47:43 maas-deb regiond[611]: yield client(
Mar 05 20:47:43 maas-deb regiond[611]: File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
Mar 05 20:47:43 maas-deb regiond[611]: current.result = callback( # type: ignore[misc]
Mar 05 20:47:43 maas-deb regiond[611]: File "/usr/lib/python3/dist-packages/provisioningserver/rpc/common.py", line 145, in _global_intercept_errback
Mar 05 20:47:43 maas-deb regiond[611]: failure.raiseException()
Mar 05 20:47:43 maas-deb regiond[611]: File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 475, in raiseException
Mar 05 20:47:43 maas-deb regiond[611]: raise self.value.with_traceback(self.tb)
Mar 05 20:47:43 maas-deb regiond[611]: provisioningserver.rpc.exceptions.CannotConfigureDHCP: DHCPv4 server failed to restart: Service 'maas-dhcpd' failed to restart. Its current state is 'SERVICE_STATE.OFF' and 'dead'.
Mar 05 20:47:43 maas-deb regiond[611]:
Mar 05 20:47:43 maas-deb regiond[611]: maasserver.dhcp: [info] Successfully configured DHCPv6 on rack controller 'maas-deb (xgbk8d)'.
Mar 05 20:47:43 maas-deb regiond[611]: maasserver.rack_controller: [critical] Failed configuring DHCP on rack controller 'id:1'.
Mar 05 20:47:43 maas-deb regiond[611]: Traceback (most recent call last):
Mar 05 20:47:43 maas-deb regiond[611]: File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1750, in gotResult
Mar 05 20:47:43 maas-deb regiond[611]: current_context.run(_inlineCallbacks, r, gen, status)
Mar 05 20:47:43 maas-deb regiond[611]: File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1740, in _inlineCallbacks
Mar 05 20:47:43 maas-deb regiond[611]: status.deferred.errback()
Mar 05 20:47:43 maas-deb regiond[611]: File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 700, in errback
Mar 05 20:47:43 maas-deb regiond[611]: self._startRunCallbacks(fail)
Mar 05 20:47:43 maas-deb regiond[611]: File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 763, in _startRunCallbacks
Mar 05 20:47:43 maas-deb regiond[611]: self._runCallbacks()
Mar 05 20:47:43 maas-deb regiond[611]: --- <exception caught here> ---
Mar 05 20:47:43 maas-deb regiond[611]: File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
Mar 05 20:47:43 maas-deb regiond[611]: current.result = callback( # type: ignore[misc]
Mar 05 20:47:43 maas-deb regiond[611]: File "/usr/lib/python3/dist-packages/maasserver/rack_controller.py", line 281, in <lambda>
Mar 05 20:47:43 maas-deb regiond[611]: d.addErrback(lambda f: f.trap(NoConnectionsAvailable))
Mar 05 20:47:43 maas-deb regiond[611]: File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 451, in trap
Mar 05 20:47:43 maas-deb regiond[611]: self.raiseException()
Mar 05 20:47:43 maas-deb regiond[611]: File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 475, in raiseException
Mar 05 20:47:43 maas-deb regiond[611]: raise self.value.with_traceback(self.tb)
Mar 05 20:47:43 maas-deb regiond[611]: File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
Mar 05 20:47:43 maas-deb regiond[611]: current.result = callback( # type: ignore[misc]
Mar 05 20:47:43 maas-deb regiond[611]: File "/usr/lib/python3/dist-packages/maasserver/rack_controller.py", line 300, in unwatch_if_does_not_exist
Mar 05 20:47:43 maas-deb regiond[611]: f.trap(RackController.DoesNotExist)
Mar 05 20:47:43 maas-deb regiond[611]: File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 451, in trap
Mar 05 20:47:43 maas-deb regiond[611]: self.raiseException()
Mar 05 20:47:43 maas-deb regiond[611]: File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 475, in raiseException
Mar 05 20:47:43 maas-deb regiond[611]: raise self.value.with_traceback(self.tb)
Mar 05 20:47:43 maas-deb regiond[611]: File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
Mar 05 20:47:43 maas-deb regiond[611]: result = current_context.run(gen.send, result)
Mar 05 20:47:43 maas-deb regiond[611]: File "/usr/lib/python3/dist-packages/maasserver/dhcp.py", line 960, in configure_dhcp
Mar 05 20:47:43 maas-deb regiond[611]: raise ipv4_exc
Mar 05 20:47:43 maas-deb regiond[611]: File "/usr/lib/python3/dist-packages/maasserver/dhcp.py", line 878, in configure_dhcp
Mar 05 20:47:43 maas-deb regiond[611]: yield client(
Mar 05 20:47:43 maas-deb regiond[611]: provisioningserver.rpc.exceptions.CannotConfigureDHCP: DHCPv4 server failed to restart: Service 'maas-dhcpd' failed to restart. Its current state is 'SERVICE_STATE.OFF' and 'dead'.
```

[Jacopo Rota (r00ta)]

Looking at

ubuntu@maas-deb35:/var/lib/maas$ sudo systemctl status maas-dhcpd
○ maas-dhcpd.service - MAAS instance of ISC DHCP server for IPv4
     Loaded: loaded (/lib/systemd/system/maas-dhcpd.service; enabled; vendor preset: enabled)
     Active: inactive (dead)
  Condition: start condition failed at Thu 2024-03-14 09:38:18 UTC; 92ms ago
             └─ ConditionPathExists=/run/maas/dhcp was not met
       Docs: man:dhcpd(8)

Mar 14 09:38:14 maas-deb35 systemd[1]: Condition check resulted in MAAS instance of ISC DHCP server for IPv4 being skipped.
Mar 14 09:38:14 maas-deb35 systemd[1]: Condition check resulted in MAAS instance of ISC DHCP server for IPv4 being skipped.
Mar 14 09:38:15 maas-deb35 systemd[1]: Condition check resulted in MAAS instance of ISC DHCP server for IPv4 being skipped.
Mar 14 09:38:15 maas-deb35 systemd[1]: Condition check resulted in MAAS instance of ISC DHCP server for IPv4 being skipped.
Mar 14 09:38:16 maas-deb35 systemd[1]: Condition check resulted in MAAS instance of ISC DHCP server for IPv4 being skipped.
Mar 14 09:38:16 maas-deb35 systemd[1]: Condition check resulted in MAAS instance of ISC DHCP server for IPv4 being skipped.
Mar 14 09:38:16 maas-deb35 systemd[1]: Condition check resulted in MAAS instance of ISC DHCP server for IPv4 being skipped.
Mar 14 09:38:17 maas-deb35 systemd[1]: Condition check resulted in MAAS instance of ISC DHCP server for IPv4 being skipped.
Mar 14 09:38:17 maas-deb35 systemd[1]: Condition check resulted in MAAS instance of ISC DHCP server for IPv4 being skipped.
Mar 14 09:38:18 maas-deb35 systemd[1]: Condition check resulted in MAAS instance of ISC DHCP server for IPv4 being skipped.
(reverse-i-search)`up': sudo apt ^Cdate
ubuntu@maas-deb35:/var/lib/maas$ cat /lib/systemd/system/maas-dhcpd.service
[Unit]
Description=MAAS instance of ISC DHCP server for IPv4
Documentation=man:dhcpd(8)
Wants=network-online.target
After=network-online.target
After=time-sync.target
BindsTo=maas-rackd.service
ConditionPathExists=/run/maas/dhcp
ConditionPathExists=/var/lib/maas/dhcpd.conf
ConditionPathExists=/var/lib/maas/dhcpd-interfaces

[Service]
# Allow dhcp server to write lease and pid file as 'dhcpd' user
# The leases files need to be root:dhcpd even when dropping privileges
ExecStart=/bin/sh -ec '\
    INTERFACES=$(cat /var/lib/maas/dhcpd-interfaces); \
    LEASES_FILE=/var/lib/maas/dhcp/dhcpd.leases; \
    [ -e $LEASES_FILE ] || touch $LEASES_FILE; \
    /usr/sbin/maas-dhcp-helper clean $LEASES_FILE; \
    chown root:dhcpd /var/lib/maas/dhcp $LEASES_FILE; \
    chmod 775 /var/lib/maas/dhcp ; chmod 664 $LEASES_FILE; \
    exec dhcpd -user dhcpd -group dhcpd -f -q -4 -pf /run/maas/dhcp/dhcpd.pid \
        -cf /var/lib/maas/dhcpd.conf -lf $LEASES_FILE $INTERFACES'
# Require dhcpd stop in 8 seconds, if not kill it with 'mixed' mode.
TimeoutStopSec=8
KillMode=mixed

[Install]
WantedBy=multi-user.target

we can see that ConditionPathExists=/run/maas/dhcp was not met . This is because we create /run/maas only when we install maas and after whatever reboot /run/maas goes away and we don't recreate it anymore (bug introduced here https://github.com/maas/maas/commit/b402c32d0a85d26fd6a5ce14a72056013fd9...