libvirt driver detach_volume fails after migration failure

[Qin Zhao]

When a VM with an attached iSCSI disk fails to migrate, the rollback methods does not detach the disk from target host. What happens is _lookup_by_name() fails, since the VM does not exist on the target host. In detach_volume(), it is supposed to print a warning based on the correct error code being returned, instead of throwing the exception. However, this is not happening, because _lookup_by_name() throws an InstanceNotFound exception, rather than a libvirt.libvirtError exception. So we also need to catch InstanceNotFound exception, so that detach_volume() can continue to execute as expected.

Here's the exception log that I have:

2014-05-16 16:30:22.328 41419 WARNING nova.compute.manager [req-3db28fed-c287-4b41-ac95-9a37a619c75c 0 4be9915c10c8426cbfe948940f7c8af1] [instance: 3e1d0d56-3370-4d05-8210-0485fa31757c] Detaching volume from unknown instance
2014-05-16 16:30:22.331 41419 ERROR nova.compute.manager [req-3db28fed-c287-4b41-ac95-9a37a619c75c 0 4be9915c10c8426cbfe948940f7c8af1] [instance: 3e1d0d56-3370-4d05-8210-0485fa31757c] Failed to detach volume 98a940e5-051f-4d0f-a8c7-859a5079d95e from /dev/vdb
2014-05-16 16:30:22.331 41419 TRACE nova.compute.manager [instance: 3e1d0d56-3370-4d05-8210-0485fa31757c] Traceback (most recent call last):
2014-05-16 16:30:22.331 41419 TRACE nova.compute.manager [instance: 3e1d0d56-3370-4d05-8210-0485fa31757c] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4218, in _detach_volume
2014-05-16 16:30:22.331 41419 TRACE nova.compute.manager [instance: 3e1d0d56-3370-4d05-8210-0485fa31757c] encryption=encryption)
2014-05-16 16:30:22.331 41419 TRACE nova.compute.manager [instance: 3e1d0d56-3370-4d05-8210-0485fa31757c] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1356, in detach_volume
2014-05-16 16:30:22.331 41419 TRACE nova.compute.manager [instance: 3e1d0d56-3370-4d05-8210-0485fa31757c] virt_dom = self._lookup_by_name(instance_name)
2014-05-16 16:30:22.331 41419 TRACE nova.compute.manager [instance: 3e1d0d56-3370-4d05-8210-0485fa31757c] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3477, in _lookup_by_name
2014-05-16 16:30:22.331 41419 TRACE nova.compute.manager [instance: 3e1d0d56-3370-4d05-8210-0485fa31757c] raise exception.InstanceNotFound(instance_id=instance_name)
2014-05-16 16:30:22.331 41419 TRACE nova.compute.manager [instance: 3e1d0d56-3370-4d05-8210-0485fa31757c] InstanceNotFound: Instance rhel65_113-3e1d0d56-00000002 could not be found.
2014-05-16 16:30:22.331 41419 TRACE nova.compute.manager [instance: 3e1d0d56-3370-4d05-8210-0485fa31757c]

[Matt Riedemann (mriedem)]

Patch is here: https://review.openstack.org/#/c/94317/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/94945

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/94317
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=52b57e82c7ef972508777a8bd3c7b937a3673d2d
Submitter: Jenkins
Branch: master

commit 52b57e82c7ef972508777a8bd3c7b937a3673d2d
Author: <email address hidden> <email address hidden>
Date: Tue May 20 13:13:26 2014 +0800

    Catch InstanceNotFound exception if migration fails

    When a VM with an attached iSCSI disk fails to migrate, the rollback methods
    does not detach the disk from target host. What happens is _lookup_by_name()
    fails, since the VM does not exist on the target host. In detach_volume(), it
    is supposed to print a warning based on the correct error code being returned,
    instead of throwing the exception. However, this is not happening, because
    _lookup_by_name() throws an InstanceNotFound exception, rather than a
    libvirt.libvirtError exception. So we also need to catch InstanceNotFound
    exception, so that detach_volume() can continue to execute as expected.

    Change-Id: I396c23d8137bf1e2834769c5483d2494506949ad
    Closes-Bug: #1321082

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/94945
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=1a45944105ecbab6870fcc4634502dd1f582d795
Submitter: Jenkins
Branch: stable/icehouse

commit 1a45944105ecbab6870fcc4634502dd1f582d795
Author: <email address hidden> <email address hidden>
Date: Tue May 20 13:13:26 2014 +0800

    Catch InstanceNotFound exception if migration fails

    When a VM with an attached iSCSI disk fails to migrate, the rollback methods
    does not detach the disk from target host. What happens is _lookup_by_name()
    fails, since the VM does not exist on the target host. In detach_volume(), it
    is supposed to print a warning based on the correct error code being returned,
    instead of throwing the exception. However, this is not happening, because
    _lookup_by_name() throws an InstanceNotFound exception, rather than a
    libvirt.libvirtError exception. So we also need to catch InstanceNotFound
    exception, so that detach_volume() can continue to execute as expected.

    Change-Id: I396c23d8137bf1e2834769c5483d2494506949ad
    Closes-Bug: #1321082
    (cherry picked from commit 52b57e82c7ef972508777a8bd3c7b937a3673d2d)