Minesweeper CI is seeing the following Tempest suites fail due to errors during performing rescue operations.
tempest.api.compute.servers.test_server_rescue.ServerRescueTestJSON
tempest.api.compute.v3.servers.test_server_rescue.ServerRescueTestXML
tempest.api.compute.v3.servers.test_server_rescue.ServerRescueV3Test
Error message seen in the nova cpu log is:
Traceback (most recent call last):
File "/opt/stack/oslo.messaging/oslo/messaging/_executors/base.py", line 36, in _dispatch
incoming.reply(self.callback(incoming.ctxt, incoming.message))
File "/opt/stack/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 134, in __call__
return self._dispatch(endpoint, method, ctxt, args)
File "/opt/stack/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 104, in _dispatch
result = getattr(endpoint, method)(ctxt, **new_args)
File "/opt/stack/nova/nova/compute/manager.py", line 356, in decorated_function
return function(self, context, *args, **kwargs)
File "/opt/stack/nova/nova/exception.py", line 88, in wrapped
payload)
File "/opt/stack/nova/nova/openstack/common/excutils.py", line 68, in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/opt/stack/nova/nova/exception.py", line 71, in wrapped
return f(self, context, *args, **kw)
File "/opt/stack/nova/nova/compute/manager.py", line 240, in decorated_function
pass
File "/opt/stack/nova/nova/openstack/common/excutils.py", line 68, in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/opt/stack/nova/nova/compute/manager.py", line 226, in decorated_function
return function(self, context, *args, **kwargs)
File "/opt/stack/nova/nova/compute/manager.py", line 291, in decorated_function
function(self, context, *args, **kwargs)
File "/opt/stack/nova/nova/compute/manager.py", line 2731, in rescue_instance
reason=_("Driver Error: %s") % unicode(e))
InstanceNotRescuable: Instance 3f317fe3-1777-4f0d-a60a-f2370d9b2fb0 cannot be rescued: Driver Error: can't set attribute
Full logs for a run that saw this error is available here: http://208.91.1.172//logs/nova/72125/1/
What's interesting is something is causing the InstanceNotResc uable fault to mask the deeper fault in vmops.py ... this could make field diagnostics very difficult. We should invest some time in another patch to address that problem. Here's the important part of the log files:
2014-02-08 14:27:04.706 DEBUG nova.virt. vmwareapi. vmops [req-725a0ee2- 276d-4252- 97dd-8e360e6572 26 ServerRescueTes tXML-107000062 ServerRescueTes tXML-601874270] [instance: 3f317fe3- 1777-4f0d- a60a-f2370d9b2f b0] Powered off the VM power_off /opt/stack/ nova/nova/ virt/vmwareapi/ vmops.py: 1152 manager [req-725a0ee2- 276d-4252- 97dd-8e360e6572 26 ServerRescueTes tXML-107000062 ServerRescueTes tXML-601874270] [instance: 3f317fe3- 1777-4f0d- a60a-f2370d9b2f b0] Error trying to Rescue Instance manager [instance: 3f317fe3- 1777-4f0d- a60a-f2370d9b2f b0] Traceback (most recent call last): manager [instance: 3f317fe3- 1777-4f0d- a60a-f2370d9b2f b0] File "/opt/stack/ nova/nova/ compute/ manager. py", line 2725, in rescue_instance manager [instance: 3f317fe3- 1777-4f0d- a60a-f2370d9b2f b0] rescue_image_meta, admin_password) manager [instance: 3f317fe3- 1777-4f0d- a60a-f2370d9b2f b0] File "/opt/stack/ nova/nova/ virt/vmwareapi/ driver. py", line 670, in rescue manager [instance: 3f317fe3- 1777-4f0d- a60a-f2370d9b2f b0] _vmops. rescue( context, instance, network_info, image_meta) manager [instance: 3f317fe3- 1777-4f0d- a60a-f2370d9b2f b0] File "/opt/stack/ nova/nova/ virt/vmwareapi/ vmops.py" , line 1087, in rescue manager [instance: 3f317fe3- 1777-4f0d- a60a-f2370d9b2f b0] r_instance['name'] = r_instance['name'] + self._rescue_suffix manager [instance: 3f317fe3- 1777-4f0d- a60a-f2370d9b2f b0] File "/opt/stack/ nova/nova/ objects/ base.py" , line 388, in __setitem__ manager [instance: 3f317fe3- 1777-4f0d- a60a-f2370d9b2f b0] setattr(self, name, value) manager [instance: 3f317fe3- 1777-4f0d- a60a-f2370d9b2f b0] AttributeError: can't set attribute manager [instance: 3f317fe3- 1777-4f0d- a60a-f2370d9b2f b0] _drivers. amqpdriver [-] MSG_ID is b979f498fc1547a 0bfff17b862ad07 77 _send /opt/stack/ oslo.messaging/ oslo/messaging/ _drivers/ amqpdriver. py:358
2014-02-08 14:27:04.708 ERROR nova.compute.
2014-02-08 14:27:04.708 24360 TRACE nova.compute.
2014-02-08 14:27:04.708 24360 TRACE nova.compute.
2014-02-08 14:27:04.708 24360 TRACE nova.compute.
2014-02-08 14:27:04.708 24360 TRACE nova.compute.
2014-02-08 14:27:04.708 24360 TRACE nova.compute.
2014-02-08 14:27:04.708 24360 TRACE nova.compute.
2014-02-08 14:27:04.708 24360 TRACE nova.compute.
2014-02-08 14:27:04.708 24360 TRACE nova.compute.
2014-02-08 14:27:04.708 24360 TRACE nova.compute.
2014-02-08 14:27:04.708 24360 TRACE nova.compute.
2014-02-08 14:27:04.708 24360 TRACE nova.compute.
2014-02-08 14:27:04.710 24360 DEBUG oslo.messaging.