XenServer: resize confirm causes 'XenAPI failed to power the instance off'

[Dan Prince]

Using the latest Nova Havana code today (git commit d4c7e88c or later)...

I'm now seeing resize_confirm API requests fail with Torpedo when running XenServer 6.1 in SmokeStack.

The following error message is displayed in Nova's compute.log file:

==> /var/log/nova/compute.log <==
2013-06-24 20:15:02.240 10813 WARNING nova.compute.manager [req-d8b83c39-614f-4861-9b2a-0d7ecadf1c77 None None] Found 0 in the database and 2 on the hypervisor.
2013-06-24 20:19:57.975 10813 WARNING nova.virt.xenapi.vm_utils [req-c58a796c-e6f8-4653-97ff-62f5340c74ca 62f8c747c81144209ca48df81a651964 1f41ab1eabef47f193a5b468001b55f3] [instance: b82e6f76-c34a-41df-a1a0-c39665a27f56] VM already halted, skipping shutdown...
2013-06-24 20:19:57.977 10813 ERROR nova.compute.manager [req-c58a796c-e6f8-4653-97ff-62f5340c74ca 62f8c747c81144209ca48df81a651964 1f41ab1eabef47f193a5b468001b55f3] [instance: b82e6f76-c34a-41df-a1a0-c39665a27f56] XenAPI failed to power the instance off. Setting instance vm_state to ERROR
2013-06-24 20:19:58.685 10813 ERROR nova.openstack.common.rpc.amqp [req-c58a796c-e6f8-4653-97ff-62f5340c74ca 62f8c747c81144209ca48df81a651964 1f41ab1eabef47f193a5b468001b55f3] Exception during message handling
2013-06-24 20:19:58.685 10813 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-06-24 20:19:58.685 10813 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/nova/openstack/common/rpc/amqp.py", line 433, in _process_data
2013-06-24 20:19:58.685 10813 TRACE nova.openstack.common.rpc.amqp **args)
2013-06-24 20:19:58.685 10813 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2013-06-24 20:19:58.685 10813 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2013-06-24 20:19:58.685 10813 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/nova/exception.py", line 99, in wrapped
2013-06-24 20:19:58.685 10813 TRACE nova.openstack.common.rpc.amqp temp_level, payload)
2013-06-24 20:19:58.685 10813 TRACE nova.openstack.common.rpc.amqp File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2013-06-24 20:19:58.685 10813 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-06-24 20:19:58.685 10813 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/nova/exception.py", line 76, in wrapped
2013-06-24 20:19:58.685 10813 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
2013-06-24 20:19:58.685 10813 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 266, in decorated_function
2013-06-24 20:19:58.685 10813 TRACE nova.openstack.common.rpc.amqp function(self, context, *args, **kwargs)
2013-06-24 20:19:58.685 10813 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 243, in decorated_function
2013-06-24 20:19:58.685 10813 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info())
2013-06-24 20:19:58.685 10813 TRACE nova.openstack.common.rpc.amqp File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2013-06-24 20:19:58.685 10813 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-06-24 20:19:58.685 10813 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 230, in decorated_function
2013-06-24 20:19:58.685 10813 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-06-24 20:19:58.685 10813 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2216, in confirm_resize
2013-06-24 20:19:58.685 10813 TRACE nova.openstack.common.rpc.amqp self._legacy_nw_info(network_info))
2013-06-24 20:19:58.685 10813 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/driver.py", line 182, in confirm_migration
2013-06-24 20:19:58.685 10813 TRACE nova.openstack.common.rpc.amqp self._vmops.confirm_migration(migration, instance, network_info)
2013-06-24 20:19:58.685 10813 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/vmops.py", line 215, in confirm_migration
2013-06-24 20:19:58.685 10813 TRACE nova.openstack.common.rpc.amqp self._destroy_orig_vm(instance, network_info)
2013-06-24 20:19:58.685 10813 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/vmops.py", line 220, in _destroy_orig_vm
2013-06-24 20:19:58.685 10813 TRACE nova.openstack.common.rpc.amqp return self._destroy(instance, vm_ref, network_info=network_info)
2013-06-24 20:19:58.685 10813 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/nova/virt/xenapi/vmops.py", line 1260, in _destroy
2013-06-24 20:19:58.685 10813 TRACE nova.openstack.common.rpc.amqp _("XenAPI failed to power the instance off"))
2013-06-24 20:19:58.685 10813 TRACE nova.openstack.common.rpc.amqp InstancePowerOffFailure: XenAPI failed to power the instance off

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/34256

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/34256
Committed: http://github.com/openstack/nova/commit/c9bb51c6da93a40f4057e99db146c109718bc3a6
Submitter: Jenkins
Branch: master

commit c9bb51c6da93a40f4057e99db146c109718bc3a6
Author: Dan Prince <email address hidden>
Date: Mon Jun 24 16:41:06 2013 -0400

    Revert XenApi virt driver should throw exception

    This reverts commit 2cb31123302a1ec8c8dec0cb2ac288a2e3b19c68.

    Fixes LP Bug #1194264.

    Change-Id: Ib8f8a8cfdcc1b37bb87fee3983081154a4fa2b2b