Instances become undelete-able if vif unplugging fails

Bug #1306718 reported by Dan Smith
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Dan Smith
Icehouse
Fix Released
High
Dan Smith

Bug Description

If an instance's vifs cannot be deleted (because, for example, they were never plugged in the first place), then compute manager will fail trying to delete the instance:

2014-04-11 09:25:27.754 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 2154, in _shutdown_instance
2014-04-11 09:25:27.754 TRACE oslo.messaging.rpc.dispatcher block_device_info)
2014-04-11 09:25:27.754 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 956, in destroy
2014-04-11 09:25:27.754 TRACE oslo.messaging.rpc.dispatcher destroy_disks)
2014-04-11 09:25:27.754 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 992, in cleanup
2014-04-11 09:25:27.754 TRACE oslo.messaging.rpc.dispatcher self.unplug_vifs(instance, network_info)
2014-04-11 09:25:27.754 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 864, in unplug_vifs
2014-04-11 09:25:27.754 TRACE oslo.messaging.rpc.dispatcher self.vif_driver.unplug(instance, vif)
2014-04-11 09:25:27.754 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/vif.py", line 783, in unplug
2014-04-11 09:25:27.754 TRACE oslo.messaging.rpc.dispatcher self.unplug_ovs(instance, vif)
2014-04-11 09:25:27.754 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/vif.py", line 667, in unplug_ovs
2014-04-11 09:25:27.754 TRACE oslo.messaging.rpc.dispatcher self.unplug_ovs_hybrid(instance, vif)
2014-04-11 09:25:27.754 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/vif.py", line 661, in unplug_ovs_hybrid
2014-04-11 09:25:27.754 TRACE oslo.messaging.rpc.dispatcher v2_name)
2014-04-11 09:25:27.754 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/network/linux_net.py", line 1317, in delete_ovs_vif_port
2014-04-11 09:25:27.754 TRACE oslo.messaging.rpc.dispatcher _ovs_vsctl(['del-port', bridge, dev])
2014-04-11 09:25:27.754 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/network/linux_net.py", line 1302, in _ovs_vsctl
2014-04-11 09:25:27.754 TRACE oslo.messaging.rpc.dispatcher raise exception.AgentError(method=full_args)
2014-04-11 09:25:27.754 TRACE oslo.messaging.rpc.dispatcher AgentError: Error during following call to agent: ['ovs-vsctl', '--timeout=120', 'del-port', 'br-int', u'qvo81ce661d-1a']

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/86957

Changed in nova:
status: Confirmed → In Progress
Dan Smith (danms)
tags: added: icehouse-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/86957
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b3225581befe14f21b909e176d0a4583b297e031
Submitter: Jenkins
Branch: master

commit b3225581befe14f21b909e176d0a4583b297e031
Author: Dan Smith <email address hidden>
Date: Fri Apr 11 09:44:27 2014 -0700

    Don't explode if we fail to unplug VIFs after a failed boot

    Libvirt currently attempts to unplug VIFs after destroying an
    instance, which can fail if the instance was never built
    properly in the first place. This patch makes it ignore errors
    when trying to tear down VIFs during that cleanup. Failure
    in this case would block the deletion of the instance (which
    should never happen) and any real failures aren't something
    we could do anything about.

    Closes-bug: #1306718

    Change-Id: I84a4433c3959b8f14e0259be0aa51da1654006c8

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
Openstack Gerrit (openstack-gerrit) wrote : Fix merged to nova (stable/icehouse)

Reviewed: https://review.openstack.org/89653
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e1e140b295ea9ea6fd6cc2b6dc10981020e48adb
Submitter: Jenkins
Branch: stable/icehouse

commit e1e140b295ea9ea6fd6cc2b6dc10981020e48adb
Author: Dan Smith <email address hidden>
Date: Fri Apr 11 09:44:27 2014 -0700

    Don't explode if we fail to unplug VIFs after a failed boot

    Libvirt currently attempts to unplug VIFs after destroying an
    instance, which can fail if the instance was never built
    properly in the first place. This patch makes it ignore errors
    when trying to tear down VIFs during that cleanup. Failure
    in this case would block the deletion of the instance (which
    should never happen) and any real failures aren't something
    we could do anything about.

    Closes-bug: #1306718

    Change-Id: I84a4433c3959b8f14e0259be0aa51da1654006c8
    (cherry picked from commit b3225581befe14f21b909e176d0a4583b297e031)

Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: juno-1 → 2014.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.