Virtual Machine could not be evacuated because virtual interface creation failed

[Matt Rabe]

I believe this issue is related to Question 257358 (https://answers.launchpad.net/ubuntu/+source/nova/+question/257358).

On the source host we see the successful vif plug:

2015-03-09 01:22:12.363 629 DEBUG neutron.plugins.ml2.rpc [req-5de70341-d64b-4a3a-bc05-54eb2802f25d None] Device 14ac5edd-269f-4808-9a34-c4cc93e9ab70 up at agent ovs-agent-ipxxxxxxxxx update_device_up /usr/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py:156
2015-03-09 01:22:12.392 629 DEBUG oslo_concurrency.lockutils [req-5de70341-d64b-4a3a-bc05-54eb2802f25d ] Acquired semaphore "db-access" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:377
2015-03-09 01:22:12.436 629 DEBUG oslo_concurrency.lockutils [req-5de70341-d64b-4a3a-bc05-54eb2802f25d ] Releasing semaphore "db-access" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:390
2015-03-09 01:22:12.437 629 DEBUG oslo_messaging._drivers.amqp [req-5de70341-d64b-4a3a-bc05-54eb2802f25d ] UNIQUE_ID is 740634ca8c7a49418a39c429669f2f27. _add_unique_id /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqp.py:224
2015-03-09 01:22:12.439 629 DEBUG oslo_messaging._drivers.amqp [req-5de70341-d64b-4a3a-bc05-54eb2802f25d ] UNIQUE_ID is 3264e8d7dd7c492d9aa17d3e9892b1fc. _add_unique_id /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqp.py:224
2015-03-09 01:22:14.436 629 DEBUG neutron.notifiers.nova [-] Sending events: [{'status': 'completed', 'tag': u'14ac5edd-269f-4808-9a34-c4cc93e9ab70', 'name': 'network-vif-plugged', 'server_uuid': u'2790be4a-5285-46aa-8ee2-c68f5b936c1d'}] send_events /usr/lib/python2.7/site-packages/neutron/notifiers/nova.py:237

Later, the destination host of the evacuation attempts to plug the vif but can't:

2015-03-09 02:15:41.441 629 DEBUG neutron.plugins.ml2.rpc [req-5ea6625c-a60c-48fb-9264-e2a5a3ed0d26 None] Device 14ac5edd-269f-4808-9a34-c4cc93e9ab70 up at agent ovs-agent-ipxxxxxxxxxx update_device_up /usr/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py:156
2015-03-09 02:15:41.485 629 DEBUG neutron.plugins.ml2.rpc [req-5ea6625c-a60c-48fb-9264-e2a5a3ed0d26 None] Device 14ac5edd-269f-4808-9a34-c4cc93e9ab70 not bound to the agent host ipxxxxxxxxx update_device_up /usr/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py:163

The cause of the problem seems to be that the neutron port does not have is binding:host_id properly updated on evacuation, the answer to question 257358 looks like the fix.

[Sean Dague (sdague)]

What version of OpenStack? what do your nova.conf and neutron.conf look like?

It would be good to know if this was already addressed in kilo, or remains a real issue in master.

[Matt Rabe (mdrabe)]

This is in kilo using the neutronv2 API (nova/network/neutronv2/api.py)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/169827

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/169827
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7c81be3c2952f26e30bb8394c8e2803d31fc3b26
Submitter: Jenkins
Branch: master

commit 7c81be3c2952f26e30bb8394c8e2803d31fc3b26
Author: Chris Friesen <email address hidden>
Date: Wed Apr 1 12:07:41 2015 -0400

    fix network setup on evacuate

    As reported in bug #1430042 when using evacuate + neutron virtual
    machines fail to evacuate because of a failure to update port
    bindings. A similar issue applies to FloatingIP setup for
    nova-network.

    A related fix was proposed in
    https://answers.launchpad.net/ubuntu/+source/nova/+question/257358
    and seemed to be fixing users.

    This handles things a bit differently in order to more cleanly
    use existing code.

    Change-Id: Iafa6d2e9eb76ea0389067c939f21ac2181df6f95
    Closes-Bug: #1430042

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/184282

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Rabe (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/184282
Reason: Wrong patch set

[Matt Rabe (mdrabe)]

I patched the fix into my environment and verified. The neutron port's binding:host_id was updated with successful evacuation. Thank you Chris

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/184282
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=1a6053b155ff7d76de271096df0f5660ed09f4d8
Submitter: Jenkins
Branch: stable/kilo

commit 1a6053b155ff7d76de271096df0f5660ed09f4d8
Author: Chris Friesen <email address hidden>
Date: Wed Apr 1 12:07:41 2015 -0400

    fix network setup on evacuate

    As reported in bug #1430042 when using evacuate + neutron virtual
    machines fail to evacuate because of a failure to update port
    bindings. A similar issue applies to FloatingIP setup for
    nova-network.

    A related fix was proposed in
    https://answers.launchpad.net/ubuntu/+source/nova/+question/257358
    and seemed to be fixing users.

    This handles things a bit differently in order to more cleanly
    use existing code.

    Change-Id: Iafa6d2e9eb76ea0389067c939f21ac2181df6f95
    Closes-Bug: #1430042
    (cherry picked from commit 7c81be3c2952f26e30bb8394c8e2803d31fc3b26)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/213195

[OpenStack Infra (hudson-openstack)]

Change abandoned by Edward Hope-Morley (<email address hidden>) on branch: stable/juno
Review: https://review.openstack.org/213195

[Corey Bryant (corey.bryant)]

This appears to also exist in Trusty so marking that as triaged.