nova network on multihost does not delete bridges during deletion of networks

[Ilya Kharin]

The teardown_unused_network_gateway parameter=true parameters is used by VlanManager in nova-network to remove bridge and vlan interfaces when the network is deleted. On Fuel 5.1.1 with CentOS 6.5 these interfaces are not deleted. In the log there are right lines:

2014-12-12 14:50:33.957 4606 DEBUG nova.openstack.common.processutils [req-8f33392b-db62-462c-a6b5-8008b7ea5412 ] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf ip link delete br204 execute /usr/lib/python2.6/site-packages/nova/openstack/common/processutils.py:154
2014-12-12 14:50:32.881 4606 DEBUG nova.network.linux_net [req-3ec984c3-fbab-45c2-9002-057a6e3306a2 None] Net device removed: 'br204' delete_net_dev /usr/lib/python2.6/site-packages/nova/network/linux_net.py:1361

But actually the bridge interface still exists:

154: br204: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
    inet 10.42.1.3/24 brd 10.42.1.255 scope global br204
    inet 10.42.2.3/24 brd 10.42.2.255 scope global br204
    inet6 fe80::109d:f0ff:feb2:9d64/64 scope link
       valid_lft forever preferred_lft forever

The command fails in the shell:

[root@node-77 ~]# ip link delete br204
RTNETLINK answers: Operation not supported
[root@node-77 ~]# echo $?
2

[Dmitry Mescheryakov (dmitrymex)]

Neutron team, the issue was reported for 5.1.1 only. Please investigate if 6.x is affected.

[Ilya Kharin (akscram)]

This bug does not affect neutron, it's a nova-network bug.

[Roman Podoliaka (rpodolyaka)]

Ok, so looks like this is not a nova-network issue, but rather a problem with linux kernel: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=680094 (ip link delete won't work, but brctl delbr works just fine)

[Roman Podoliaka (rpodolyaka)]

Overall, I don't think this should be treated as High, as routing works as expected except the fact we have a couple of unused bridges hanging.

If we really want to see this fixed, we should either patch the kernel or hack nova-network (use brctl delbr instead of ip link delete+ ensure we've got a corresponding rootwrap rule). Neither of the solutions is going to be implemented in the maintenance release.

[Roman Podoliaka (rpodolyaka)]

5.1+ ubuntu is not affected

[Roman Podoliaka (rpodolyaka)]

Only CentOS is affected due to using of 2.6.32 kernel.

[Roman Podoliaka (rpodolyaka)]

So the fix will be to do:

  sudo ip link set <dev> down
  sudo brctl delbr <dev>

for bridge devices here

https://github.com/openstack/nova/blob/stable/juno/nova/network/linux_net.py#L1617-L1639

[Roman Podoliaka (rpodolyaka)]

^ is a workaround for older kernels

[Dmitry Mescheryakov (dmitrymex)]

This is a medium issue, so we are not going to fix it in a maintenance release

[OSCI Robot (oscirobot)]

package nova has been built for project openstack/nova
Package version == 2014.2.2, package release == fuel6.1.mira2.git.025759a.bea7645

Changeset: https://review.fuel-infra.org/3595
project: openstack/nova
branch: openstack-ci/fuel-6.1/2014.2
author: Marian Horban
committer: Marian Horban
subject: Deleting of bridge corrected
status: patchset-created

Files placed on repository:

NOTE: Changeset is not merged, created temporary package repository.
 repository URL: /centos-fuel-6.1-stable-3595/

[OSCI Robot (oscirobot)]

package nova has been built for project openstack/nova
Package version == 2014.2.2, package release == fuel6.1~mira2+git.025759a.bea7645

Changeset: https://review.fuel-infra.org/3595
project: openstack/nova
branch: openstack-ci/fuel-6.1/2014.2
author: Marian Horban
committer: Marian Horban
subject: Deleting of bridge corrected
status: patchset-created

Files placed on repository:

NOTE: Changeset is not merged, created temporary package repository.
 repository URL: /ubuntu-fuel-6.1-stable-3595/

[Alexander Gubanov (ogubanov)]

I've verified it on MOS6.1 (build 432) - fixed!
ENV: CentOS - HA - Nova network VLAN - 3 controllers, 2 computes with Cinder
Proof here: http://pastebin.com/iquFQ6LE

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Marian Horban <email address hidden>
Review: https://review.fuel-infra.org/8256

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Marian Horban <email address hidden>
Review: https://review.fuel-infra.org/8257

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Kholkin Pavel <email address hidden> on branch: openstack-ci/fuel-7.0/2015.1.0
Review: https://review.fuel-infra.org/8257
Reason: This patch is already in Kilo

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Kholkin Pavel <email address hidden> on branch: openstack-ci/fuel-7.0/2015.1.0
Review: https://review.fuel-infra.org/8256
Reason: This patch is already in Kilo