[2.3.6] selecting pod VM in gui returns "'MachineHandler' object has no attribute 'dehydrate_pod'"

Bug #1835480 reported by Xav Paice on 2019-07-05
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
MAAS
Undecided
Unassigned
2.3
High
Adam Collard

Bug Description

Recently upgraded a Xenial cluster of 3 Maas hosts from 2.3.3 to 2.3.6 using apt. The nodes are 10.35.101.10, 10.35.101.11, and 10.35.101.12 with VIP 10.35.101.31 for Postgres, 10.35.101.32 for Maas.

Previously, I was able to view Pod vms in the web UI, but now when I click on any pod composed VM from the 'Nodes' tab, the window returns "'MachineHandler' object has no attribute 'dehydrate_pod'". In order to escape that, I need to click on another tab and refresh the browser with a ctrl-shift-r.

I've tried restarting the maas services on each host, plus Apache2.

When I select the Pods tab, and a particular machine, I get a panel showing me that there's 8 composed machines, but the list where the VMs usually appear is blank. At the same time, in regiond.log, I see:

Traceback (most recent call last):
Failure: twisted.internet.error.MulticastJoinError: (b'\xe0\x00\x00v', b'\n#e\x0b', 98, 'Address already in use')
2019-07-05 06:11:22 regiond: [info] 10.35.101.12 GET /MAAS/rpc/ HTTP/1.0 --> 200 OK (referrer: -; agent: provisioningserver.rpc.clusterservice.ClusterClientService)
2019-07-05 06:11:23 maasserver.models.signals.interfaces: [info] eno3 (physical) on openstack-2: VLAN updated to match bond1 (bond) on openstack-2 (vlan=None).
2019-07-05 06:11:24 maasserver.models.signals.interfaces: [info] eno4 (physical) on openstack-2: VLAN updated to match bond1 (bond) on openstack-2 (vlan=None).
2019-07-05 06:11:24 regiond: [info] 10.35.101.12 GET /MAAS/rpc/ HTTP/1.0 --> 200 OK (referrer: -; agent: provisioningserver.rpc.clusterservice.ClusterClientService)
2019-07-05 06:11:25 provisioningserver.utils.services: [critical] Failed to update and/or record network interface configuration: 'NoneType' object has no attribute 'fabric'; interfaces: {'bond2': {'source': 'ipaddr', 'mac_address': '0c:c4:7a:2a:eb:6b', 'enabled': True, 'links': [], 'type': 'bond', 'parents': ['eno4', 'eno3'], 'index': 9, 'monitored': True}, 'eno2': {'source': 'ipaddr', 'mac_address': '0c:c4:7a:2a:eb:69', 'enabled': False, 'links': [], 'type': 'physical', 'parents': [], 'index': 5, 'monitored': False}, 'virbr0': {'source': 'ipaddr', 'mac_address': '52:54:00:0f:7d:16', 'enabled': True, 'links': [{'mode': 'static', 'address': '192.168.122.1/24'}], 'type': 'bridge', 'parents': [], 'index': 13, 'monitored': True}, 'eno1': {'source': 'ipaddr', 'mac_address': '0c:c4:7a:2a:eb:68', 'enabled': False, 'links': [], 'type': 'physical', 'parents': [], 'index': 4, 'monitored': False}, 'bond2.1107': {'source': 'ipaddr', 'vid': 1107, 'enabled': True, 'links': [{'mode': 'static', 'address': '10.35.107.251/24'}], 'type': 'vlan', 'parents': ['bond2'], 'index': 11, 'monitored': False}, 'ens2f0': {'source': 'ipaddr', 'mac_address': 'ac:1f:6b:24:62:74', 'enabled': True, 'links': [], 'type': 'physical', 'parents': [], 'index': 2, 'monitored': False}, 'broam': {'source': 'ipaddr', 'mac_address': 'ac:1f:6b:24:62:74', 'enabled': True, 'links': [{'mode': 'static', 'address': '10.35.101.11/24', 'gateway': '10.35.101.1'}, {'mode': 'static', 'address': '10.35.101.31/24', 'gateway': '10.35.101.1'}], 'type': 'bridge', 'parents': ['bond0'], 'index': 10, 'monitored': False}, 'ens2f1': {'source': 'ipaddr', 'mac_address': 'ac:1f:6b:24:62:75', 'enabled': True, 'links': [], 'type': 'physical', 'parents': [], 'index': 3, 'monitored': False}, 'bond0': {'source': 'ipaddr', 'mac_address': 'ac:1f:6b:24:62:74', 'enabled': True, 'links': [], 'type': 'bond', 'parents': ['ens2f0', 'ens2f1'], 'index': 8, 'monitored': True}, 'eno4': {'source': 'ipaddr', 'mac_address': '0c:c4:7a:2a:eb:6b', 'enabled': True, 'links': [], 'type': 'physical', 'parents': [], 'index': 7, 'monitored': False}, 'eno3': {'source': 'ipaddr', 'mac_address': '0c:c4:7a:2a:eb:6a', 'enabled': True, 'links': [], 'type': 'physical', 'parents': [], 'index': 6, 'monitored': False}, 'bond1': {'source': 'ipaddr', 'mac_address': 'ce:c2:e8:3f:6e:08', 'enabled': True, 'links': [], 'type': 'bond', 'parents': [], 'index': 12, 'monitored': True}}

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 588, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1184, in gotResult
    _inlineCallbacks(r, g, deferred)
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
--- <exception caught here> ---
  File "/usr/lib/python3/dist-packages/provisioningserver/utils/services.py", line 1000, in updateInterfaces
    yield self._updateInterfaces(interfaces)
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/lib/python3/dist-packages/provisioningserver/utils/services.py", line 1112, in _updateInterfaces
    yield maybeDeferred(self.recordInterfaces, interfaces, hints)
  File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 246, in inContext
    result = inContext.theWork()
  File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 262, in <lambda>
    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
  File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
  File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 875, in callInContext
    return func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 232, in wrapper
    result = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 754, in call_within_transaction
    return func_outside_txn(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 561, in retrier
    return func(*args, **kwargs)
  File "/usr/lib/python3.5/contextlib.py", line 30, in inner
    return func(*args, **kwds)
  File "/usr/lib/python3/dist-packages/maasserver/regiondservices/networks_monitoring.py", line 49, in recordInterfacesIntoDatabase
    region_controller.update_interfaces(interfaces, hints)
  File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 232, in wrapper
    result = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 710, in call_with_connection
    return func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/maasserver/utils/__init__.py", line 192, in call_with_lock
    return func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 738, in call_within_transaction
    return func_within_txn(*args, **kwargs)
  File "/usr/lib/python3.5/contextlib.py", line 30, in inner
    return func(*args, **kwds)
  File "/usr/lib/python3/dist-packages/maasserver/models/node.py", line 4906, in update_interfaces
    hints=topology_hints)
  File "/usr/lib/python3/dist-packages/maasserver/models/node.py", line 4279, in _update_interface
    return self._update_vlan_interface(name, config)
  File "/usr/lib/python3/dist-packages/maasserver/models/node.py", line 4500, in _update_vlan_interface
    fabric=parent_nic.vlan.fabric, vid=config["vid"])
builtins.AttributeError: 'NoneType' object has no attribute 'fabric'
2019-07-05 06:11:25 regiond: [info] 10.35.101.12 GET /MAAS/rpc/ HTTP/1.0 --> 200 OK (referrer: -; agent: provisioningserver.rpc.clusterservice.ClusterClientService)

Ignoring the gui, I tried to compose a fresh machine and commission it via the CLI, but the commission is failing with just a timeout, as if pxe boot isn't even working.

Related branches

Revision history for this message
Newell Jensen (newell-jensen) wrote :

2.3.6 is no longer supported, can you reproduce this in 2.6.1?

Changed in maas:
status: New → Incomplete
Revision history for this message
Janghoon-Paul Sim (janghoon) wrote :

@Newwell,
I'm sorry, Maas v2.3.6 is no longer supported?
v2.3 is still in Xenial repository and I thought it's supported until EOD of Xenial.
Please correct me if I'm wrong.

Revision history for this message
Xav Paice (xavpaice) wrote :

This customer is running MaaS 2.3.6:

maas-common:
  Installed: 2.3.6-6526-g182d293-0ubuntu1~16.04.1
  Candidate: 2.3.6-6526-g182d293-0ubuntu1~16.04.1
  Version table:
 *** 2.3.6-6526-g182d293-0ubuntu1~16.04.1 500
        500 http://ppa.launchpad.net/maas/proposed/ubuntu xenial/main amd64 Packages
        500 http://ppa.launchpad.net/maas/proposed/ubuntu xenial/main i386 Packages
        500 http://ppa.launchpad.net/maas/stable/ubuntu xenial/main amd64 Packages
        500 http://ppa.launchpad.net/maas/stable/ubuntu xenial/main i386 Packages
        100 /var/lib/dpkg/status
     2.3.5-6511-gf466fdb-0ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
        500 http://archive.ubuntu.com/ubuntu xenial-updates/main i386 Packages
     2.0.0~beta3+bzr4941-0ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages
        500 http://archive.ubuntu.com/ubuntu xenial/main i386 Packages

2.3.6 is ahead of the Xenial release 2.3.5, are you saying that we should try to downgrade to 2.3.5, or upgrade to something else? I don't see Xenial packages in the 2.6 PPA.

Changed in maas:
status: Incomplete → Opinion
status: Opinion → New
Revision history for this message
Xav Paice (xavpaice) wrote :

I've subscribed canonical-field-medium because this affects a current customer and is preventing us from deploying services there.

Revision history for this message
Dongwon Cho (dongwoncho) wrote :

I have done the following test but wasn't able to reproduce it.
- Setup a new MAAS cluster without PPA
- Checked the WEB UI was opening but didn't touch anything except that.
- Upgraded from 2.3.5 to 2.3.6 adding PPA
- Login the WEB UI and checked [Nodes] -> [Controllers] -> 1st controller

I guess, there might be something wrong with the data migration.

Revision history for this message
Björn Tillenius (bjornt) wrote :

For the bug that is mentioned in the title (MachineHandler has no dehydrate_pod()), I can confirm that this is broken in 2.3.6. Commit 182d29396be7019615dd3eaa8f8afda0d33d86cc added some code that is obviously untested. From the commit message it looks to be a bad cherry-pick and that code shouldn't be there at all.

As for the error you're seeing in regiond.log, could you please file a separate bug for that? That will make things easier to track.

Changed in maas:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers