DHCP errors after upgrade to 3.3.4

[James Ssutherland]

After an upgrade from MAAS 3.2.8 to 3.3.4 (including upgrade form Ubuntu 20 to Ubuntu 22 and Postgres 12 to Postgres 14) region log is full of these errors. DHCP is working, but in the controller screen, dhcpd goes red, then green, and cycles over and over. Region sometimes fails to load its GUI for several minutes, then starts working again.

2023-12-04 23:26:43 maasserver.dhcp: [info] Successfully configured DHCPv6 on rack controller 'Z5CEINT1BM020201-MAAS-001 (fyepk8)'.
2023-12-04 23:26:43 maasserver.rack_controller: [critical] Failed configuring DHCP on rack controller 'id:622'.
        Traceback (most recent call last):
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1750, in gotResult
            current_context.run(_inlineCallbacks, r, gen, status)
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1740, in _inlineCallbacks
            status.deferred.errback()
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 700, in errback
            self._startRunCallbacks(fail)
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 763, in _startRunCallbacks
            self._runCallbacks()
        --- <exception caught here> ---
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
            current.result = callback( # type: ignore[misc]
          File "/usr/lib/python3/dist-packages/maasserver/rack_controller.py", line 281, in <lambda>
            d.addErrback(lambda f: f.trap(NoConnectionsAvailable))
          File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 451, in trap
            self.raiseException()
          File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 475, in raiseException
            raise self.value.with_traceback(self.tb)
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
            current.result = callback( # type: ignore[misc]
          File "/usr/lib/python3/dist-packages/maasserver/rack_controller.py", line 300, in unwatch_if_does_not_exist
            f.trap(RackController.DoesNotExist)
          File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 451, in trap
            self.raiseException()
          File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 475, in raiseException
            raise self.value.with_traceback(self.tb)
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
            result = current_context.run(gen.send, result)
          File "/usr/lib/python3/dist-packages/maasserver/dhcp.py", line 954, in configure_dhcp
            raise ipv4_exc
          File "/usr/lib/python3/dist-packages/maasserver/dhcp.py", line 872, in configure_dhcp
            yield client(
        twisted.internet.defer.CancelledError:

2023-12-04 23:26:49 maasserver.dhcp: [critical] Error configuring DHCPv4 on rack controller 'Z5CEINT1BM020201-MAAS-001 (fyepk8)':
        Traceback (most recent call last):
        --- <exception caught here> ---
          File "/usr/lib/python3/dist-packages/maasserver/dhcp.py", line 872, in configure_dhcp
            yield client(
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
            current.result = callback( # type: ignore[misc]
          File "/usr/lib/python3/dist-packages/twisted/protocols/amp.py", line 1946, in _massageError
            error.trap(RemoteAmpError)
          File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 451, in trap
            self.raiseException()
          File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 475, in raiseException
            raise self.value.with_traceback(self.tb)
        twisted.internet.defer.CancelledError:

[Christian Grabowski (cgrabowski)]

Hi there! Can you please attach any logs from that time? Or preferably an sosreport if possible?

[James Ssutherland (jamesesutherland)]

Here is an SOS report

[James Ssutherland (jamesesutherland)]

We got this working again by adding the exact string listed as the host name in MAAS of the rack/region controller to the 127.0.1.1 line of /etc/hosts.

There seems to be something different with Ubuntu 22 or MAAS 3.8 in how it makes RPC connections. It uses a name name instead of an IP, maybe.

The line in /etc/hosts looks like this now (we added the last entry)
127.0.1.1 Z5CEINT1BM020201.CEC Z5CEINT1BM020201 Z5CEINT1BM020201-MAAS-001

[Thorsten Merten (thorsten-merten)]

Does the workaround do the right thing for you? Does this need attention and is still a problem when upgrading to newer versions?

[Launchpad Janitor (janitor)]

[Expired for MAAS 3.3 because there has been no activity for 60 days.]