Recently upgraded a Xenial cluster of 3 Maas hosts from 2.3.3 to 2.3.6 using apt. The nodes are 10.35.101.10, 10.35.101.11, and 10.35.101.12 with VIP 10.35.101.31 for Postgres, 10.35.101.32 for Maas.
Previously, I was able to view Pod vms in the web UI, but now when I click on any pod composed VM from the 'Nodes' tab, the window returns "'MachineHandler' object has no attribute 'dehydrate_pod'". In order to escape that, I need to click on another tab and refresh the browser with a ctrl-shift-r.
I've tried restarting the maas services on each host, plus Apache2.
When I select the Pods tab, and a particular machine, I get a panel showing me that there's 8 composed machines, but the list where the VMs usually appear is blank. At the same time, in regiond.log, I see:
Traceback (most recent call last):
Failure: twisted.internet.error.MulticastJoinError: (b'\xe0\x00\x00v', b'\n#e\x0b', 98, 'Address already in use')
2019-07-05 06:11:22 regiond: [info] 10.35.101.12 GET /MAAS/rpc/ HTTP/1.0 --> 200 OK (referrer: -; agent: provisioningserver.rpc.clusterservice.ClusterClientService)
2019-07-05 06:11:23 maasserver.models.signals.interfaces: [info] eno3 (physical) on openstack-2: VLAN updated to match bond1 (bond) on openstack-2 (vlan=None).
2019-07-05 06:11:24 maasserver.models.signals.interfaces: [info] eno4 (physical) on openstack-2: VLAN updated to match bond1 (bond) on openstack-2 (vlan=None).
2019-07-05 06:11:24 regiond: [info] 10.35.101.12 GET /MAAS/rpc/ HTTP/1.0 --> 200 OK (referrer: -; agent: provisioningserver.rpc.clusterservice.ClusterClientService)
2019-07-05 06:11:25 provisioningserver.utils.services: [critical] Failed to update and/or record network interface configuration: 'NoneType' object has no attribute 'fabric'; interfaces: {'bond2': {'source': 'ipaddr', 'mac_address': '0c:c4:7a:2a:eb:6b', 'enabled': True, 'links': [], 'type': 'bond', 'parents': ['eno4', 'eno3'], 'index': 9, 'monitored': True}, 'eno2': {'source': 'ipaddr', 'mac_address': '0c:c4:7a:2a:eb:69', 'enabled': False, 'links': [], 'type': 'physical', 'parents': [], 'index': 5, 'monitored': False}, 'virbr0': {'source': 'ipaddr', 'mac_address': '52:54:00:0f:7d:16', 'enabled': True, 'links': [{'mode': 'static', 'address': '192.168.122.1/24'}], 'type': 'bridge', 'parents': [], 'index': 13, 'monitored': True}, 'eno1': {'source': 'ipaddr', 'mac_address': '0c:c4:7a:2a:eb:68', 'enabled': False, 'links': [], 'type': 'physical', 'parents': [], 'index': 4, 'monitored': False}, 'bond2.1107': {'source': 'ipaddr', 'vid': 1107, 'enabled': True, 'links': [{'mode': 'static', 'address': '10.35.107.251/24'}], 'type': 'vlan', 'parents': ['bond2'], 'index': 11, 'monitored': False}, 'ens2f0': {'source': 'ipaddr', 'mac_address': 'ac:1f:6b:24:62:74', 'enabled': True, 'links': [], 'type': 'physical', 'parents': [], 'index': 2, 'monitored': False}, 'broam': {'source': 'ipaddr', 'mac_address': 'ac:1f:6b:24:62:74', 'enabled': True, 'links': [{'mode': 'static', 'address': '10.35.101.11/24', 'gateway': '10.35.101.1'}, {'mode': 'static', 'address': '10.35.101.31/24', 'gateway': '10.35.101.1'}], 'type': 'bridge', 'parents': ['bond0'], 'index': 10, 'monitored': False}, 'ens2f1': {'source': 'ipaddr', 'mac_address': 'ac:1f:6b:24:62:75', 'enabled': True, 'links': [], 'type': 'physical', 'parents': [], 'index': 3, 'monitored': False}, 'bond0': {'source': 'ipaddr', 'mac_address': 'ac:1f:6b:24:62:74', 'enabled': True, 'links': [], 'type': 'bond', 'parents': ['ens2f0', 'ens2f1'], 'index': 8, 'monitored': True}, 'eno4': {'source': 'ipaddr', 'mac_address': '0c:c4:7a:2a:eb:6b', 'enabled': True, 'links': [], 'type': 'physical', 'parents': [], 'index': 7, 'monitored': False}, 'eno3': {'source': 'ipaddr', 'mac_address': '0c:c4:7a:2a:eb:6a', 'enabled': True, 'links': [], 'type': 'physical', 'parents': [], 'index': 6, 'monitored': False}, 'bond1': {'source': 'ipaddr', 'mac_address': 'ce:c2:e8:3f:6e:08', 'enabled': True, 'links': [], 'type': 'bond', 'parents': [], 'index': 12, 'monitored': True}}
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 588, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1184, in gotResult
_inlineCallbacks(r, g, deferred)
File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
--- <exception caught here> ---
File "/usr/lib/python3/dist-packages/provisioningserver/utils/services.py", line 1000, in updateInterfaces
yield self._updateInterfaces(interfaces)
File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/usr/lib/python3/dist-packages/provisioningserver/utils/services.py", line 1112, in _updateInterfaces
yield maybeDeferred(self.recordInterfaces, interfaces, hints)
File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 246, in inContext
result = inContext.theWork()
File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 262, in <lambda>
inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 118, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 81, in callWithContext
return func(*args,**kw)
File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 875, in callInContext
return func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 232, in wrapper
result = func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 754, in call_within_transaction
return func_outside_txn(*args, **kwargs)
File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 561, in retrier
return func(*args, **kwargs)
File "/usr/lib/python3.5/contextlib.py", line 30, in inner
return func(*args, **kwds)
File "/usr/lib/python3/dist-packages/maasserver/regiondservices/networks_monitoring.py", line 49, in recordInterfacesIntoDatabase
region_controller.update_interfaces(interfaces, hints)
File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 232, in wrapper
result = func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 710, in call_with_connection
return func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/maasserver/utils/__init__.py", line 192, in call_with_lock
return func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 738, in call_within_transaction
return func_within_txn(*args, **kwargs)
File "/usr/lib/python3.5/contextlib.py", line 30, in inner
return func(*args, **kwds)
File "/usr/lib/python3/dist-packages/maasserver/models/node.py", line 4906, in update_interfaces
hints=topology_hints)
File "/usr/lib/python3/dist-packages/maasserver/models/node.py", line 4279, in _update_interface
return self._update_vlan_interface(name, config)
File "/usr/lib/python3/dist-packages/maasserver/models/node.py", line 4500, in _update_vlan_interface
fabric=parent_nic.vlan.fabric, vid=config["vid"])
builtins.AttributeError: 'NoneType' object has no attribute 'fabric'
2019-07-05 06:11:25 regiond: [info] 10.35.101.12 GET /MAAS/rpc/ HTTP/1.0 --> 200 OK (referrer: -; agent: provisioningserver.rpc.clusterservice.ClusterClientService)
Ignoring the gui, I tried to compose a fresh machine and commission it via the CLI, but the commission is failing with just a timeout, as if pxe boot isn't even working.
2.3.6 is no longer supported, can you reproduce this in 2.6.1?