Linux bridge and vethpair were not cleaned after instance is removed

[Vitalii]

After I added/removed routers, nets and subnets many times, for testing purpose, I found that I have 45 interfaces:

b# ifconfig|grep encap:Ethernet
br-eth0 Link encap:Ethernet HWaddr d0:67:e5:03:18:0f
br-eth0:1 Link encap:Ethernet HWaddr d0:67:e5:03:18:0f
eth0 Link encap:Ethernet HWaddr d0:67:e5:03:18:0f
int-br-eth0 Link encap:Ethernet HWaddr 82:ad:59:16:ca:da
phy-br-eth0 Link encap:Ethernet HWaddr da:e4:8d:cd:38:43
qbr056025bc-49 Link encap:Ethernet HWaddr fa:c1:6d:fe:5e:b8
qbr3dccedf9-e8 Link encap:Ethernet HWaddr 02:42:c0:73:d2:0b
qbr422898c8-df Link encap:Ethernet HWaddr 76:b0:42:f5:ad:f9
qbr69eb1f6a-71 Link encap:Ethernet HWaddr d6:d2:7f:ee:1a:34
qbr750aa557-b7 Link encap:Ethernet HWaddr 5a:2e:b3:f3:30:b1
qbr81eb2deb-b7 Link encap:Ethernet HWaddr f2:d0:95:19:fb:c4
qbr971c890b-8f Link encap:Ethernet HWaddr ce:a1:c9:f0:b6:24
qbr9ab03868-2f Link encap:Ethernet HWaddr 2a:d5:8a:61:0a:a1
qbrcfd38872-d1 Link encap:Ethernet HWaddr 16:7d:ad:1b:4b:82
qbrde55f70b-7d Link encap:Ethernet HWaddr 2a:8c:f5:2b:49:99
qbrdead1da5-98 Link encap:Ethernet HWaddr d6:95:61:73:1d:c6
qbrf0db8340-a6 Link encap:Ethernet HWaddr 9e:9b:36:c3:cb:d3
qbrf3f3c43f-ff Link encap:Ethernet HWaddr 4a:c6:5c:70:e4:b4
qvb056025bc-49 Link encap:Ethernet HWaddr fa:c1:6d:fe:5e:b8
qvb3dccedf9-e8 Link encap:Ethernet HWaddr 02:42:c0:73:d2:0b
qvb422898c8-df Link encap:Ethernet HWaddr 76:b0:42:f5:ad:f9
qvb69eb1f6a-71 Link encap:Ethernet HWaddr d6:d2:7f:ee:1a:34
qvb750aa557-b7 Link encap:Ethernet HWaddr 5a:2e:b3:f3:30:b1
qvb81eb2deb-b7 Link encap:Ethernet HWaddr f2:d0:95:19:fb:c4
qvb971c890b-8f Link encap:Ethernet HWaddr ce:a1:c9:f0:b6:24
qvb9ab03868-2f Link encap:Ethernet HWaddr 2a:d5:8a:61:0a:a1
qvbcfd38872-d1 Link encap:Ethernet HWaddr 16:7d:ad:1b:4b:82
qvbde55f70b-7d Link encap:Ethernet HWaddr 2a:8c:f5:2b:49:99
qvbdead1da5-98 Link encap:Ethernet HWaddr d6:95:61:73:1d:c6
qvbf0db8340-a6 Link encap:Ethernet HWaddr 9e:9b:36:c3:cb:d3
qvbf3f3c43f-ff Link encap:Ethernet HWaddr 4a:c6:5c:70:e4:b4
qvo056025bc-49 Link encap:Ethernet HWaddr aa:82:8b:9f:d6:a0
qvo3dccedf9-e8 Link encap:Ethernet HWaddr ea:8c:1f:0e:ab:92
qvo422898c8-df Link encap:Ethernet HWaddr 7a:9f:47:3c:3b:57
qvo69eb1f6a-71 Link encap:Ethernet HWaddr a6:dd:41:ce:e6:39
qvo750aa557-b7 Link encap:Ethernet HWaddr 32:6c:f8:ca:af:e9
qvo81eb2deb-b7 Link encap:Ethernet HWaddr ea:22:94:19:ac:4c
qvo971c890b-8f Link encap:Ethernet HWaddr 2e:f8:a7:72:1c:85
qvo9ab03868-2f Link encap:Ethernet HWaddr aa:3e:bb:c6:6d:d3
qvocfd38872-d1 Link encap:Ethernet HWaddr 16:3a:12:30:f5:71
qvode55f70b-7d Link encap:Ethernet HWaddr fa:ee:28:ed:52:37
qvodead1da5-98 Link encap:Ethernet HWaddr 5a:66:51:d9:a5:60
qvof0db8340-a6 Link encap:Ethernet HWaddr 66:b6:23:c9:ca:73
qvof3f3c43f-ff Link encap:Ethernet HWaddr 5e:5b:53:e8:11:58
tapdead1da5-98 Link encap:Ethernet HWaddr fe:54:00:39:74:0a

Also ther'a 265 namespaces.

I use Icehouse.

[Jakub Libosvar (libosvar)]

What you see there are bridges and its veth pairs for instances' security groups and connection between tap device and ovs. AFAIK vif driver in nova should care about its deletions and these devices are not created in case you create new router but rather instances.

[yalei wang (yalei-wang)]

could you show your port-list net-list info, looks like there still ports there.

[Vitalii (vb-d)]

I know, I learned neutron architecture before submitting bugs. By the way, having 9 interfaces, that packet should pass on its way to/from VM is a classical example of overengineering. I'm sure there's more neat solution.

port-list shown only ports of router and one VM, in total about 5-6 ports.

[Eugene Nikanorov (enikanorov)]

Not sure what exactly the issue is.

Are you talking about number of interfaces/ports not match?
Please provide more specific data about the issue.

Also note that if you have been taking snapshots during your test runs, there could be some lags between port operations and their reflection on physical host.

Marking as Incomplete

[Vitalii (vb-d)]

I did not do snapshots.

The issue is that it becomes difficult to find needed namespace, veth device and maintain system with such amount of records everywhere.

[Zengfa Gao (zfgao)]

I saw similar issue with my dev environment after I unstack and stack many times.
root@zegacn1:/home/zega# ifconfig | grep encap
br-int Link encap:Ethernet HWaddr 3e:af:f8:3a:3d:40
br-tun Link encap:Ethernet HWaddr 76:26:6e:24:05:43
eth0 Link encap:Ethernet HWaddr 00:0c:29:d8:3b:ab
eth1 Link encap:Ethernet HWaddr 00:0c:29:d8:3b:b5
eth2 Link encap:Ethernet HWaddr 00:0c:29:d8:3b:bf
lo Link encap:Local Loopback
qbr03a4f8d6-e5 Link encap:Ethernet HWaddr 7a:9a:91:94:d5:d7
qbr1e564f70-f1 Link encap:Ethernet HWaddr 5a:20:4b:28:08:ec
qbrbd6bd7f0-c7 Link encap:Ethernet HWaddr 16:20:c0:8f:3f:ad
qbrdcdc77d2-c6 Link encap:Ethernet HWaddr 4e:95:ed:c2:1f:3d
qvb03a4f8d6-e5 Link encap:Ethernet HWaddr 7a:9a:91:94:d5:d7
qvb1e564f70-f1 Link encap:Ethernet HWaddr 5a:20:4b:28:08:ec
qvbbd6bd7f0-c7 Link encap:Ethernet HWaddr 16:20:c0:8f:3f:ad
qvbdcdc77d2-c6 Link encap:Ethernet HWaddr 4e:95:ed:c2:1f:3d
qvo03a4f8d6-e5 Link encap:Ethernet HWaddr b6:fd:4a:5f:86:de
qvo1e564f70-f1 Link encap:Ethernet HWaddr 3e:ee:1f:63:a8:4b
qvobd6bd7f0-c7 Link encap:Ethernet HWaddr 96:a0:49:0e:86:31
qvodcdc77d2-c6 Link encap:Ethernet HWaddr 5a:bb:e4:cc:bc:c8
tap1e564f70-f1 Link encap:Ethernet HWaddr fe:16:3e:6f:03:03
virbr0 Link encap:Ethernet HWaddr 36:96:e8:ad:2b:9f

In neutron database, there is no port.
zega@zegacn1:~/devstack$ neutron port-list
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
| id | name | mac_address | fixed_ips |
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
| 1e564f70-f121-48ad-af68-99ea38419248 | | fa:16:3e:6f:03:03 | {"subnet_id": "fc07a567-c83e-4ec3-9015-2ceb71485759", "ip_address": "10.0.0.2"} |
| 73e02e4e-12c2-4dcf-843d-631d813768e0 | | fa:16:3e:a2:09:44 | {"subnet_id": "fc07a567-c83e-4ec3-9015-2ceb71485759", "ip_address": "10.0.0.3"} |
| 76e80323-258d-4e90-b12f-aa6971fae156 | | fa:16:3e:6a:46:c8 | {"subnet_id": "fc07a567-c83e-4ec3-9015-2ceb71485759", "ip_address": "10.0.0.1"} |
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+

I think OVS created the port during stack.sh, but didn't cleanup during unstack.sh. Something with devstack?

[Cedric Brandily (cbrandily)]

these interfaces qvoXXX/qvbXXX/qbrXXX/tapXXX are created by nova (when ovs_hybrid_plug==True) not neutron.

[Daniel Berrange (berrange)]

What virt driver are you using with Nova ?

[Vitalii (vb-d)]

compute_driver = nova.virt.libvirt.LibvirtDriver
vif_driver = nova.virt.libvirt.vif.NeutronLinuxBridgeVIFDriver
use_virtio_for_bridges=True

[Sean Dague (sdague)]

is this actually really a Neutron issue?

[Eugene Nikanorov (enikanorov)]

I don't find enough information here to consider this as a valid Neutron issue.

[Vitalii (vb-d)]

Maybe just try to use Openstack withtout devstack ?

This issue happens on all systems where I use Openstack. I create networks, then create instances, then remove networks. All with API, NOT COMMAND LINE.

Every time I submitted bug report you were asking me how to reproduce it from command line. I almost do not use CLI.

Guys, I would like to help Openstack to become stable, but you want me to put too much efforts to help you reproduce issues.

[Wu Hong Guang (whg)]

reproduce step :

1: install devstack with neutron enabled
2:login in demo and go to project/network and make sue there are
   2 network :public and private
3:launch a instance and attach to private network then assign a floating ip
4: uninstall devstack by run ./unstack.sh and ./clean.sh

there are many qbrXXX/qvrXXX/qvoXXX left uncleaned

[Wu Hong Guang (whg)]

delet qbr manually :

sudo ip link set qbrXXXX down
sudo brctl delbr qbrXXXX

[Sean Dague (sdague)]

this is definitely neutron leakage, removing nova

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]