Activity log for bug #1792031

Date Who What changed Old value New value Message
2018-09-11 20:56:33 Andres Rodriguez bug added bug
2018-09-11 20:56:52 Andres Rodriguez description 2018-09-11 20:55:45 maasserver.dhcp: [info] Successfully configured DHCPv6 on rack controller 'ycqan6'. 2018-09-11 20:55:45 maasserver.rack_controller: [critical] Failed configuring DHCP on rack controller 'id:1'. Traceback (most recent call last): File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1442, in gotResult _inlineCallbacks(r, g, deferred) File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1432, in _inlineCallbacks deferred.errback() File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 500, in errback self._startRunCallbacks(fail) File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 567, in _startRunCallbacks self._runCallbacks() --- <exception caught here> --- File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 653, in _runCallbacks current.result = callback(current.result, *args, **kw) File "/usr/lib/python3/dist-packages/maasserver/rack_controller.py", line 256, in <lambda> d.addErrback(lambda f: f.trap(NoConnectionsAvailable)) File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 359, in trap self.raiseException() File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 385, in raiseException raise self.value.with_traceback(self.tb) File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks result = g.send(result) File "/usr/lib/python3/dist-packages/maasserver/dhcp.py", line 835, in configure_dhcp raise ipv4_exc File "/usr/lib/python3/dist-packages/maasserver/dhcp.py", line 773, in configure_dhcp omapi_key=config.omapi_key) provisioningserver.rpc.exceptions.CannotConfigureDHCP: timed out On a multi-node MAAS I started seeing the following: 2018-09-11 20:55:45 maasserver.dhcp: [info] Successfully configured DHCPv6 on rack controller 'ycqan6'. 2018-09-11 20:55:45 maasserver.rack_controller: [critical] Failed configuring DHCP on rack controller 'id:1'.         Traceback (most recent call last):           File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1442, in gotResult             _inlineCallbacks(r, g, deferred)           File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1432, in _inlineCallbacks             deferred.errback()           File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 500, in errback             self._startRunCallbacks(fail)           File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 567, in _startRunCallbacks             self._runCallbacks()         --- <exception caught here> ---           File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 653, in _runCallbacks             current.result = callback(current.result, *args, **kw)           File "/usr/lib/python3/dist-packages/maasserver/rack_controller.py", line 256, in <lambda>             d.addErrback(lambda f: f.trap(NoConnectionsAvailable))           File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 359, in trap             self.raiseException()           File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 385, in raiseException             raise self.value.with_traceback(self.tb)           File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks             result = g.send(result)           File "/usr/lib/python3/dist-packages/maasserver/dhcp.py", line 835, in configure_dhcp             raise ipv4_exc           File "/usr/lib/python3/dist-packages/maasserver/dhcp.py", line 773, in configure_dhcp             omapi_key=config.omapi_key)         provisioningserver.rpc.exceptions.CannotConfigureDHCP: timed out
2018-09-11 20:56:55 Andres Rodriguez maas: milestone 2.5.0beta2
2018-09-11 20:56:56 Andres Rodriguez maas: importance Undecided Critical
2018-09-11 20:56:58 Andres Rodriguez maas: status New Triaged
2018-09-11 20:57:22 Andres Rodriguez description On a multi-node MAAS I started seeing the following: 2018-09-11 20:55:45 maasserver.dhcp: [info] Successfully configured DHCPv6 on rack controller 'ycqan6'. 2018-09-11 20:55:45 maasserver.rack_controller: [critical] Failed configuring DHCP on rack controller 'id:1'.         Traceback (most recent call last):           File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1442, in gotResult             _inlineCallbacks(r, g, deferred)           File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1432, in _inlineCallbacks             deferred.errback()           File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 500, in errback             self._startRunCallbacks(fail)           File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 567, in _startRunCallbacks             self._runCallbacks()         --- <exception caught here> ---           File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 653, in _runCallbacks             current.result = callback(current.result, *args, **kw)           File "/usr/lib/python3/dist-packages/maasserver/rack_controller.py", line 256, in <lambda>             d.addErrback(lambda f: f.trap(NoConnectionsAvailable))           File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 359, in trap             self.raiseException()           File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 385, in raiseException             raise self.value.with_traceback(self.tb)           File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks             result = g.send(result)           File "/usr/lib/python3/dist-packages/maasserver/dhcp.py", line 835, in configure_dhcp             raise ipv4_exc           File "/usr/lib/python3/dist-packages/maasserver/dhcp.py", line 773, in configure_dhcp             omapi_key=config.omapi_key)         provisioningserver.rpc.exceptions.CannotConfigureDHCP: timed out On a multi-node MAAS I started seeing the following. I dont have an specific way to reproduce, but just keep seeing this from time to time. 2018-09-11 20:55:45 maasserver.dhcp: [info] Successfully configured DHCPv6 on rack controller 'ycqan6'. 2018-09-11 20:55:45 maasserver.rack_controller: [critical] Failed configuring DHCP on rack controller 'id:1'.         Traceback (most recent call last):           File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1442, in gotResult             _inlineCallbacks(r, g, deferred)           File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1432, in _inlineCallbacks             deferred.errback()           File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 500, in errback             self._startRunCallbacks(fail)           File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 567, in _startRunCallbacks             self._runCallbacks()         --- <exception caught here> ---           File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 653, in _runCallbacks             current.result = callback(current.result, *args, **kw)           File "/usr/lib/python3/dist-packages/maasserver/rack_controller.py", line 256, in <lambda>             d.addErrback(lambda f: f.trap(NoConnectionsAvailable))           File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 359, in trap             self.raiseException()           File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 385, in raiseException             raise self.value.with_traceback(self.tb)           File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks             result = g.send(result)           File "/usr/lib/python3/dist-packages/maasserver/dhcp.py", line 835, in configure_dhcp             raise ipv4_exc           File "/usr/lib/python3/dist-packages/maasserver/dhcp.py", line 773, in configure_dhcp             omapi_key=config.omapi_key)         provisioningserver.rpc.exceptions.CannotConfigureDHCP: timed out
2018-09-11 21:00:22 Andres Rodriguez description On a multi-node MAAS I started seeing the following. I dont have an specific way to reproduce, but just keep seeing this from time to time. 2018-09-11 20:55:45 maasserver.dhcp: [info] Successfully configured DHCPv6 on rack controller 'ycqan6'. 2018-09-11 20:55:45 maasserver.rack_controller: [critical] Failed configuring DHCP on rack controller 'id:1'.         Traceback (most recent call last):           File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1442, in gotResult             _inlineCallbacks(r, g, deferred)           File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1432, in _inlineCallbacks             deferred.errback()           File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 500, in errback             self._startRunCallbacks(fail)           File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 567, in _startRunCallbacks             self._runCallbacks()         --- <exception caught here> ---           File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 653, in _runCallbacks             current.result = callback(current.result, *args, **kw)           File "/usr/lib/python3/dist-packages/maasserver/rack_controller.py", line 256, in <lambda>             d.addErrback(lambda f: f.trap(NoConnectionsAvailable))           File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 359, in trap             self.raiseException()           File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 385, in raiseException             raise self.value.with_traceback(self.tb)           File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks             result = g.send(result)           File "/usr/lib/python3/dist-packages/maasserver/dhcp.py", line 835, in configure_dhcp             raise ipv4_exc           File "/usr/lib/python3/dist-packages/maasserver/dhcp.py", line 773, in configure_dhcp             omapi_key=config.omapi_key)         provisioningserver.rpc.exceptions.CannotConfigureDHCP: timed out On a multi-node MAAS I started seeing the following. I dont have an specific way to reproduce, but just keep seeing this from time to time. Note that the UI shows, on the rack controller, shows a failure (dhcpd - timed out) when it fails, but a few seconds after, it fixes itself, another seconds later, it fails again. So this is intermitent. 2018-09-11 20:55:45 maasserver.dhcp: [info] Successfully configured DHCPv6 on rack controller 'ycqan6'. 2018-09-11 20:55:45 maasserver.rack_controller: [critical] Failed configuring DHCP on rack controller 'id:1'.         Traceback (most recent call last):           File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1442, in gotResult             _inlineCallbacks(r, g, deferred)           File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1432, in _inlineCallbacks             deferred.errback()           File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 500, in errback             self._startRunCallbacks(fail)           File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 567, in _startRunCallbacks             self._runCallbacks()         --- <exception caught here> ---           File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 653, in _runCallbacks             current.result = callback(current.result, *args, **kw)           File "/usr/lib/python3/dist-packages/maasserver/rack_controller.py", line 256, in <lambda>             d.addErrback(lambda f: f.trap(NoConnectionsAvailable))           File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 359, in trap             self.raiseException()           File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 385, in raiseException             raise self.value.with_traceback(self.tb)           File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks             result = g.send(result)           File "/usr/lib/python3/dist-packages/maasserver/dhcp.py", line 835, in configure_dhcp             raise ipv4_exc           File "/usr/lib/python3/dist-packages/maasserver/dhcp.py", line 773, in configure_dhcp             omapi_key=config.omapi_key)         provisioningserver.rpc.exceptions.CannotConfigureDHCP: timed out
2018-09-12 01:00:49 Blake Rouse maas: importance Critical High
2018-09-17 08:23:01 Andres Rodriguez maas: assignee Blake Rouse (blake-rouse)
2018-09-17 10:01:00 Andres Rodriguez tags sprint track
2018-09-18 13:02:10 Blake Rouse merge proposal linked https://code.launchpad.net/~blake-rouse/maas/+git/maas/+merge/355204
2018-09-18 13:02:15 Blake Rouse maas: status Triaged In Progress
2018-09-18 14:16:26 MAAS Lander maas: status In Progress Fix Committed
2018-10-03 15:01:57 Andres Rodriguez maas: status Fix Committed Fix Released