Virtual Machine could not be evacuated because virtual interface creation failed

Bug #1430042 reported by Matt Rabe on 2015-03-09
36
This bug affects 4 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Medium
Chris Friesen
Kilo
Undecided
Unassigned
Ubuntu Cloud Archive
Undecided
Unassigned
Kilo
Undecided
Unassigned
Liberty
Undecided
Unassigned
Mitaka
Undecided
Unassigned
Ubuntu
Undecided
Unassigned
Trusty
Medium
Unassigned

Bug Description

I believe this issue is related to Question 257358 (https://answers.launchpad.net/ubuntu/+source/nova/+question/257358).

On the source host we see the successful vif plug:

2015-03-09 01:22:12.363 629 DEBUG neutron.plugins.ml2.rpc [req-5de70341-d64b-4a3a-bc05-54eb2802f25d None] Device 14ac5edd-269f-4808-9a34-c4cc93e9ab70 up at agent ovs-agent-ipxxxxxxxxx update_device_up /usr/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py:156
2015-03-09 01:22:12.392 629 DEBUG oslo_concurrency.lockutils [req-5de70341-d64b-4a3a-bc05-54eb2802f25d ] Acquired semaphore "db-access" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:377
2015-03-09 01:22:12.436 629 DEBUG oslo_concurrency.lockutils [req-5de70341-d64b-4a3a-bc05-54eb2802f25d ] Releasing semaphore "db-access" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:390
2015-03-09 01:22:12.437 629 DEBUG oslo_messaging._drivers.amqp [req-5de70341-d64b-4a3a-bc05-54eb2802f25d ] UNIQUE_ID is 740634ca8c7a49418a39c429669f2f27. _add_unique_id /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqp.py:224
2015-03-09 01:22:12.439 629 DEBUG oslo_messaging._drivers.amqp [req-5de70341-d64b-4a3a-bc05-54eb2802f25d ] UNIQUE_ID is 3264e8d7dd7c492d9aa17d3e9892b1fc. _add_unique_id /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqp.py:224
2015-03-09 01:22:14.436 629 DEBUG neutron.notifiers.nova [-] Sending events: [{'status': 'completed', 'tag': u'14ac5edd-269f-4808-9a34-c4cc93e9ab70', 'name': 'network-vif-plugged', 'server_uuid': u'2790be4a-5285-46aa-8ee2-c68f5b936c1d'}] send_events /usr/lib/python2.7/site-packages/neutron/notifiers/nova.py:237

Later, the destination host of the evacuation attempts to plug the vif but can't:

2015-03-09 02:15:41.441 629 DEBUG neutron.plugins.ml2.rpc [req-5ea6625c-a60c-48fb-9264-e2a5a3ed0d26 None] Device 14ac5edd-269f-4808-9a34-c4cc93e9ab70 up at agent ovs-agent-ipxxxxxxxxxx update_device_up /usr/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py:156
2015-03-09 02:15:41.485 629 DEBUG neutron.plugins.ml2.rpc [req-5ea6625c-a60c-48fb-9264-e2a5a3ed0d26 None] Device 14ac5edd-269f-4808-9a34-c4cc93e9ab70 not bound to the agent host ipxxxxxxxxx update_device_up /usr/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py:163

The cause of the problem seems to be that the neutron port does not have is binding:host_id properly updated on evacuation, the answer to question 257358 looks like the fix.

Sean Dague (sdague) wrote :

What version of OpenStack? what do your nova.conf and neutron.conf look like?

It would be good to know if this was already addressed in kilo, or remains a real issue in master.

Changed in nova:
status: New → Incomplete
Matt Rabe (mdrabe) wrote :

This is in kilo using the neutronv2 API (nova/network/neutronv2/api.py)

Sean Dague (sdague) on 2015-04-01
tags: added: neutron

Fix proposed to branch: master
Review: https://review.openstack.org/169827

Changed in nova:
assignee: nobody → Sean Dague (sdague)
status: Incomplete → In Progress
Sean Dague (sdague) on 2015-04-01
Changed in nova:
importance: Undecided → Medium
Chris Friesen (cbf123) on 2015-04-15
tags: added: kilo-backport-potential
Changed in nova:
assignee: Sean Dague (sdague) → Chris Friesen (cbf123)

Reviewed: https://review.openstack.org/169827
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7c81be3c2952f26e30bb8394c8e2803d31fc3b26
Submitter: Jenkins
Branch: master

commit 7c81be3c2952f26e30bb8394c8e2803d31fc3b26
Author: Chris Friesen <email address hidden>
Date: Wed Apr 1 12:07:41 2015 -0400

    fix network setup on evacuate

    As reported in bug #1430042 when using evacuate + neutron virtual
    machines fail to evacuate because of a failure to update port
    bindings. A similar issue applies to FloatingIP setup for
    nova-network.

    A related fix was proposed in
    https://answers.launchpad.net/ubuntu/+source/nova/+question/257358
    and seemed to be fixing users.

    This handles things a bit differently in order to more cleanly
    use existing code.

    Change-Id: Iafa6d2e9eb76ea0389067c939f21ac2181df6f95
    Closes-Bug: #1430042

Changed in nova:
status: In Progress → Fix Committed

Change abandoned by Matt Rabe (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/184282
Reason: Wrong patch set

Matt Rabe (mdrabe) wrote :

I patched the fix into my environment and verified. The neutron port's binding:host_id was updated with successful evacuation. Thank you Chris

Reviewed: https://review.openstack.org/184282
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=1a6053b155ff7d76de271096df0f5660ed09f4d8
Submitter: Jenkins
Branch: stable/kilo

commit 1a6053b155ff7d76de271096df0f5660ed09f4d8
Author: Chris Friesen <email address hidden>
Date: Wed Apr 1 12:07:41 2015 -0400

    fix network setup on evacuate

    As reported in bug #1430042 when using evacuate + neutron virtual
    machines fail to evacuate because of a failure to update port
    bindings. A similar issue applies to FloatingIP setup for
    nova-network.

    A related fix was proposed in
    https://answers.launchpad.net/ubuntu/+source/nova/+question/257358
    and seemed to be fixing users.

    This handles things a bit differently in order to more cleanly
    use existing code.

    Change-Id: Iafa6d2e9eb76ea0389067c939f21ac2181df6f95
    Closes-Bug: #1430042
    (cherry picked from commit 7c81be3c2952f26e30bb8394c8e2803d31fc3b26)

tags: added: in-stable-kilo
Thierry Carrez (ttx) on 2015-06-24
Changed in nova:
milestone: none → liberty-1
status: Fix Committed → Fix Released

Change abandoned by Edward Hope-Morley (<email address hidden>) on branch: stable/juno
Review: https://review.openstack.org/213195

Thierry Carrez (ttx) on 2015-10-15
Changed in nova:
milestone: liberty-1 → 12.0.0
James Page (james-page) on 2016-09-08
Changed in cloud-archive:
status: New → Invalid
Corey Bryant (corey.bryant) wrote :

This appears to also exist in Trusty so marking that as triaged.

Changed in ubuntu:
status: New → Triaged
status: Triaged → Incomplete
status: Incomplete → Fix Released
status: Fix Released → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Related questions