VIFs unplugged twice during VM restart?

[dan wendlandt]

When I try reboot a VM, the nova-compute fails and the daemon stop.
It fails on unpluging VIF interface of the rebooted VM. I use Nova
with Quantum and OVS.

2012-03-01 18:12:31 DEBUG nova.rpc.common [-] Making asynchronous call
on network ... from (pid=21485) multicall
/usr/local/src/nova/nova/rpc/amqp.py:318
2012-03-01 18:12:31 DEBUG nova.rpc.common [-] MSG_ID is
b77f0f284af045f396b90fddd16f724e from (pid=21485) multicall
/usr/local/src/nova/nova/rpc/amqp.py:321
2012-03-01 18:12:33 DEBUG nova.utils [-] Running cmd (subprocess):
sudo nova-rootwrap ovs-vsctl del-port br-int tap9f5ef54c-38 from
(pid=21485) execute /usr/local/src/nova/nova/utils.py:208
2012-03-01 18:12:33 DEBUG nova.utils [-] Running cmd (subprocess):
sudo nova-rootwrap ip link delete tap9f5ef54c-38 from (pid=21485)
execute /usr/local/src/nova/nova/utils.py:208
2012-03-01 18:12:34 DEBUG nova.utils [-] Running cmd (subprocess):
sudo nova-rootwrap ovs-vsctl del-port br-int tap285c8465-dd from
(pid=21485) execute /usr/local/src/nova/nova/utils.py:208
2012-03-01 18:12:34 DEBUG nova.utils [-] Running cmd (subprocess):
sudo nova-rootwrap ip link delete tap285c8465-dd from (pid=21485)
execute /usr/local/src/nova/nova/utils.py:208
2012-03-01 18:12:34 DEBUG nova.utils [-] Attempting to grab semaphore
"iptables" for method "apply"... from (pid=21485) inner
/usr/local/src/nova/nova/utils.py:824
2012-03-01 18:12:34 DEBUG nova.utils [-] Got semaphore "iptables" for
method "apply"... from (pid=21485) inner
/usr/local/src/nova/nova/utils.py:828
2012-03-01 18:12:34 DEBUG nova.utils [-] Attempting to grab file lock
"iptables" for method "apply"... from (pid=21485) inner
/usr/local/src/nova/nova/utils.py:832
2012-03-01 18:12:34 DEBUG nova.utils [-] Got file lock "iptables" for
method "apply"... from (pid=21485) inner
/usr/local/src/nova/nova/utils.py:839
2012-03-01 18:12:34 DEBUG nova.utils [-] Running cmd (subprocess):
sudo nova-rootwrap iptables-save -t filter from (pid=21485) execute
/usr/local/src/nova/nova/utils.py:208
2012-03-01 18:12:35 INFO nova.virt.libvirt.connection [-] [instance:
9300df8d-1a47-40ce-b90e-28f1a84f064d] Instance destroyed successfully.
2012-03-01 18:12:35 DEBUG nova.utils [-] Running cmd (subprocess):
sudo nova-rootwrap iptables-restore from (pid=21485) execute
/usr/local/src/nova/nova/utils.py:208
2012-03-01 18:12:35 DEBUG nova.utils [-] Running cmd (subprocess):
sudo nova-rootwrap iptables-save -t nat from (pid=21485) execute
/usr/local/src/nova/nova/utils.py:208
2012-03-01 18:12:35 DEBUG nova.utils [-] Running cmd (subprocess):
sudo nova-rootwrap iptables-restore from (pid=21485) execute
/usr/local/src/nova/nova/utils.py:208
2012-03-01 18:12:35 DEBUG nova.network.linux_net [-]
IPTablesManager.apply completed with success from (pid=21485) apply
/usr/local/src/nova/nova/network/linux_net.py:336
2012-03-01 18:12:37 DEBUG nova.utils [-] Running cmd (subprocess):
sudo nova-rootwrap ovs-vsctl del-port br-int tap9f5ef54c-38 from
(pid=21485) execute /usr/local/src/nova/nova/utils.py:208
2012-03-01 18:12:37 DEBUG nova.utils [-] Result was 1 from (pid=21485)
execute /usr/local/src/nova/nova/utils.py:224
2012-03-01 18:12:37 WARNING nova.virt.libvirt.vif [-] Failed while
unplugging vif of instance 'instance-00000002'

[dan wendlandt (danwent)]

At a glance, it seems that unplug() being called twice, with the second call failing because the first call successfully unplugged the vif already.

The unplug() action on the bridge does nothing, so there's no harm in calling it twice, which may explain why this is only seen with OVS.

A quick workaround would be to remove the "raise" line at the end of the unplug() method for LibvirtOpenVswitchDriver, in which case you will see the log message still, but it should not disrupt the reboot (this is probably better behavior as well, regardless of the true cause of the bug).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/4754

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/4754
Committed: http://github.com/openstack/nova/commit/04e57f169646dff5309177ce5dfa444f668bf8a1
Submitter: Jenkins
Branch: master

commit 04e57f169646dff5309177ce5dfa444f668bf8a1
Author: Dan Wendlandt <email address hidden>
Date: Thu Mar 1 11:52:33 2012 -0800

    libvirt driver calls unplug() twice on vm reboot.

    bug 944168

    Also prevents OVS vif plugging driver from raising an exception
    if deleting the OVS port fails. We already log an error and
    print the exception, and there's
    no reason that failing to delete the OVS port should cause the whole operation
    to fail.

    Change-Id: I01c22ab8627762831a106b407f758f6592363f33