evacuate on ceph backed volume fails

[Blane Bramble]

When using nova evacuate to move an instance from one compute host to another, the command silently fails. The issue seems to be that the rebuild process builds an incorrect libvirt.xml file that no longer correctly references the ceph volume.

Specifically under the <disk> section I see:

<source protocol="rbd" name="volumes/instance-00000004_disk">

where in the original libvirt.xml the file was:

<source protocol="rbd" name="volumes/volume-9e1a7835-b780-495c-a88a-4558be784dde">

[Blane Bramble (blane)]

The problem seems to be as follows:

During the rebuild prep_block_device calls _prep_block_device - this in turn calls DriverVolumeBlockDevice with the existing bdms as the argument - this bdms does not seem to have source_type or destination_type set, and so the call fails.

[Blane Bramble (blane)]

As a work-around, changing the relevant block_device_mapping_get_all_by_instance call to not use legacy mode seems to work:

[Matt Thompson (mattt416)]

I see the same thing on Havana -- Blane, which version are you running?

[Blane Bramble (blane)]

Hi Matt, yes this is with Havana.

[Bart Wensley (bartwensley)]

The same problem occurs when evacuating an instance on a volume with an NFS backend - the libvirt.xml file on the destination compute host is built incorrectly and references a disk image instead of the cinder volume. Testing done in Havana release.

I applied the patch Blane provided and that seems to fix the problem.

[Andres Isaac Benavides (andres-i-benavides)]

The patch that Blane provided is working ok with Ceph Backend also.

[Andres Isaac Benavides (andres-i-benavides)]

Blane just a question, Could we this patch as a solution for this bug?

[Blane Bramble (blane)]

Hi Andres, of course.

[Bart Wensley (bartwensley)]

Andres,

Before you fix this, it would be good to understand if any of the other calls to block_device_mapping_get_all_by_instance in nova.compute.manager.py that do not set legacy=False are also broken. If they are, they should also be fixed. I don't have a good enough understanding of this code myself to make that determination, but perhaps someone who is following this bug can comment?

Bart

[Andres Isaac Benavides (andres-i-benavides)]

Yes Bart, I am going to investigate the other calls in nova.compute.manager.py

[Andres Isaac Benavides (andres-i-benavides)]

I did not find anything unusual respect to other calls in nova.compute.manager. I do not know if someone who is following this bug can comment?

[Ruel Masalta (rmasalta)]

Hi,

anyone can help me for the details on how to do the tweak?

--- old/manager.py 2013-11-12 15:21:23.824525122 +0000
+++ new/manager.py 2013-11-12 17:11:07.952673903 +0000
@@ -2021,7 +2021,7 @@
             if bdms is None:
                 bdms = self.conductor_api.\
                         block_device_mapping_get_all_by_instance(
- context, instance)
+ context, instance, False)

             # NOTE(sirp): this detach is necessary b/c we will reattach the
             # volumes in _prep_block_devices below.

[Ruel Masalta (rmasalta)]

I've seen a lot of "if bdms is None:" blocks in manager.py file. Do I have to modify all of them?

[Ruel Masalta (rmasalta)]

Having a hard time applying the workaround in icehouse. Any help would be much appreciated.

[Jakub Pavlik (pavlk-jakub)]

I confirm this bug. Patch works for IBM SVC driver also.

[Dmitry Mescheryakov (dmitrymex)]

The bug is mirrored in MOS there: https://bugs.launchpad.net/mos/+bug/1367610

[Dmitry Borodaenko (angdraug)]

There's a related (potentially duplicate) bug #1340411.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/131613

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/121745
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=91d3272b975572d9866b7d959547e438142dc4fb
Submitter: Jenkins
Branch: master

commit 91d3272b975572d9866b7d959547e438142dc4fb
Author: Fei Long Wang <email address hidden>
Date: Tue Sep 16 15:43:37 2014 +1200

    Fix nova evacuate issues for RBD

    For RBD scenario, there are some issues in Nova code
    now against evacuate function:

    1. Based on current implementation, nova evacuate and
    nova rebuild are sharing some code. When user enables
    the on_shared_storage option for nova evacuate, nova
    will check if the instance path is accessible. For
    the RBD scenario, the volume(block) is shared between
    different hosts, though the path isn't shared at the
    filesystem level. This patch fixes this issue and adds
    test cases for that.

    2. Missing the 'recreate' parameter for rebuild method.
    Though the libvirt driver doesn't implement rebuild
    method(only Ironic driver implements it), but we really
    need to set 'recreate' in kwargs so it gets passed to
    _rebuild_default_impl so we don't call driver.destroy
    on evacuate for shared filesystem/block storage cases.
    It is fixed in this patch and test case is added as well.

    Closes-Bug: 1249319
    Closes-Bug: 1340411

    Change-Id: Idc8c45b055e986cf85730235d5d25777632ad1c1

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/131629

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/131613
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7920cfdab2fb10e01544eeb713a1e3bc79bc4996
Submitter: Jenkins
Branch: stable/juno

commit 7920cfdab2fb10e01544eeb713a1e3bc79bc4996
Author: Fei Long Wang <email address hidden>
Date: Tue Sep 16 15:43:37 2014 +1200

    Fix nova evacuate issues for RBD

    For RBD scenario, there are some issues in Nova code
    now against evacuate function:

    1. Based on current implementation, nova evacuate and
    nova rebuild are sharing some code. When user enables
    the on_shared_storage option for nova evacuate, nova
    will check if the instance path is accessible. For
    the RBD scenario, the volume(block) is shared between
    different hosts, though the path isn't shared at the
    filesystem level. This patch fixes this issue and adds
    test cases for that.

    2. Missing the 'recreate' parameter for rebuild method.
    Though the libvirt driver doesn't implement rebuild
    method(only Ironic driver implements it), but we really
    need to set 'recreate' in kwargs so it gets passed to
    _rebuild_default_impl so we don't call driver.destroy
    on evacuate for shared filesystem/block storage cases.
    It is fixed in this patch and test case is added as well.

    Closes-Bug: 1249319
    Closes-Bug: 1340411

    Change-Id: Idc8c45b055e986cf85730235d5d25777632ad1c1
    (cherry picked from commit 91d3272b975572d9866b7d959547e438142dc4fb)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/131629
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3de3f1066fa47312b8c3075abf790631034d67a3
Submitter: Jenkins
Branch: stable/icehouse

commit 3de3f1066fa47312b8c3075abf790631034d67a3
Author: Fei Long Wang <email address hidden>
Date: Tue Sep 16 15:43:37 2014 +1200

    Fix nova evacuate issues for RBD

    For RBD scenario, there are some issues in Nova code
    now against evacuate function:

    1. Based on current implementation, nova evacuate and
    nova rebuild are sharing some code. When user enables
    the on_shared_storage option for nova evacuate, nova
    will check if the instance path is accessible. For
    the RBD scenario, the volume(block) is shared between
    different hosts, though the path isn't shared at the
    filesystem level. This patch fixes this issue and adds
    test cases for that.

    2. Missing the 'recreate' parameter for rebuild method.
    Though the libvirt driver doesn't implement rebuild
    method(only Ironic driver implements it), but we really
    need to set 'recreate' in kwargs so it gets passed to
    _rebuild_default_impl so we don't call driver.destroy
    on evacuate for shared filesystem/block storage cases.
    It is fixed in this patch and test case is added as well.

    Closes-Bug: 1249319
    Closes-Bug: 1340411

    Conflicts:
            nova/tests/compute/test_compute_mgr.py
            nova/tests/virt/libvirt/test_libvirt.py
            nova/virt/libvirt/driver.py

    Change-Id: Idc8c45b055e986cf85730235d5d25777632ad1c1
    (cherry picked from commit 91d3272b975572d9866b7d959547e438142dc4fb)
    (cherry picked from commit 7920cfdab2fb10e01544eeb713a1e3bc79bc4996)