OpenStack Compute (nova)

libvirt: Trying to delete a non-existing vif raises an exception

Bug #1308544 reported by Phil Day
38
This bug affects 6 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
Phil Day
OpenStack Compute (nova) 2014.2 "juno"
Icehouse
Fix Released
Undecided
Unassigned
OpenStack Compute (nova) 2014.1.4

Bug Description

If an instance fails during its network creation (for example if the network-vif-plugged event doesn't arrive in time) a subsequent delete will also fail when it tries to delete the vif, leaving the instance in a Error(deleting) state.

This can be avoided by including the "--if-exists" option to the ovs=vsctl command.

Example of stack trace:

 2014-04-16 12:28:51.949 AUDIT nova.compute.manager [req-af72c100-5d9b-44f6-b941-3d72529b3401 demo demo] [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] Terminating instance
2014-04-16 12:28:52.309 ERROR nova.virt.libvirt.driver [-] [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] During wait destroy, instance disappeared.
2014-04-16 12:28:52.407 ERROR nova.network.linux_net [req-af72c100-5d9b-44f6-b941-3d72529b3401 demo demo] Unable to execute ['ovs-vsctl', '--timeout=120', 'del-port', 'br-int', u'qvo67a96e96-10']. Exception: Unexpected error while running command.
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf ovs-vsctl --timeout=120 del-port br-int qvo67a96e96-10
Exit code: 1
Stdout: ''
Stderr: 'ovs-vsctl: no port named qvo67a96e96-10\n'
2014-04-16 12:28:52.573 ERROR nova.compute.manager [req-af72c100-5d9b-44f6-b941-3d72529b3401 demo demo] [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] Setting instance vm_state to ERROR
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] Traceback (most recent call last):
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/compute/manager.py", line 2261, in do_terminate_instance
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] self._delete_instance(context, instance, bdms, quotas)
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/hooks.py", line 103, in inner
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] rv = f(*args, **kwargs)
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/compute/manager.py", line 2231, in _delete_instance
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] quotas.rollback()
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] six.reraise(self.type_, self.value, self.tb)
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/compute/manager.py", line 2203, in _delete_instance
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] self._shutdown_instance(context, db_inst, bdms)
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/compute/manager.py", line 2145, in _shutdown_instance
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] requested_networks)
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] six.reraise(self.type_, self.value, self.tb)
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/compute/manager.py", line 2135, in _shutdown_instance
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] block_device_info)
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/virt/libvirt/driver.py", line 955, in destroy
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] destroy_disks)
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/virt/libvirt/driver.py", line 991, in cleanup
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] self.unplug_vifs(instance, network_info)
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/virt/libvirt/driver.py", line 863, in unplug_vifs
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] self.vif_driver.unplug(instance, vif)
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/virt/libvirt/vif.py", line 783, in unplug
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] self.unplug_ovs(instance, vif)
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/virt/libvirt/vif.py", line 667, in unplug_ovs
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] self.unplug_ovs_hybrid(instance, vif)
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/virt/libvirt/vif.py", line 661, in unplug_ovs_hybrid
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] v2_name)
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/network/linux_net.py", line 1318, in delete_ovs_vif_port
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] _ovs_vsctl(['del-port', bridge, dev])
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] File "/mnt/stack/nova/nova/network/linux_net.py", line 1302, in _ovs_vsctl
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] raise exception.AgentError(method=full_args)
2014-04-16 12:28:52.573 TRACE nova.compute.manager [instance: 3b7ac090-1ada-4beb-9e56-1ba3a6445e1f] AgentError: Error during following call to agent: ['ovs-vsctl', '--timeout=120', 'del-port', 'br-int', u'qvo67a96e96-10']

Phil Day (philip-day)
Changed in nova:
assignee: nobody → Phil Day (philip-day)
Revision history for this message
Openstack Gerrit (openstack-gerrit) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/87998

Changed in nova:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/87998
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=476dc5efcdb11c9859b062dc69a8c205e69be861
Submitter: Jenkins
Branch: master

commit 476dc5efcdb11c9859b062dc69a8c205e69be861
Author: Phil Day <email address hidden>
Date: Wed Apr 16 15:23:15 2014 +0000

    Ignore errors when deleting non-existing vifs

    If an instance fails during its network creation (for example
    if the network-vif-plugged event doesn't arrive in time) a
    subsequent delete will also fail when it tries to delete the
    vif, leaving the instance in a Error(deleting) state.

    This can be avoided by including the "--if-exists" option to
    the ovs-vsctl command so that the command doesn't fail if the
    vif doesn't exist.

    Change-Id: Ic613c9d6e0619f51fcd931e62f6d6bce404ebe81
    Closes-bug: #1308544

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → juno-1
status: Fix Committed → Fix Released
Jay Pipes (jaypipes)
tags: added: icehouse-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/icehouse)

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/125210

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/icehouse)

Reviewed: https://review.openstack.org/125210
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=698c821ad7b6878041be15901e89ddf6dd2c1636
Submitter: Jenkins
Branch: stable/icehouse

commit 698c821ad7b6878041be15901e89ddf6dd2c1636
Author: Phil Day <email address hidden>
Date: Wed Apr 16 15:23:15 2014 +0000

    Ignore errors when deleting non-existing vifs

    If an instance fails during its network creation (for example
    if the network-vif-plugged event doesn't arrive in time) a
    subsequent delete will also fail when it tries to delete the
    vif, leaving the instance in a Error(deleting) state.

    This can be avoided by including the "--if-exists" option to
    the ovs-vsctl command so that the command doesn't fail if the
    vif doesn't exist.

    Change-Id: Ic613c9d6e0619f51fcd931e62f6d6bce404ebe81
    Closes-bug: #1308544
    (cherry picked from commit 476dc5efcdb11c9859b062dc69a8c205e69be861)

tags: added: in-stable-icehouse
Thierry Carrez (ttx)
Changed in nova:
milestone: juno-1 → 2014.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.