neutron multiple external flat networks fails

[bjolo]

Setting up one external flat network works fine. However when trying to setup more than one external flat network, neutron-openvswitch-agent disconnects ovs continously. This bug will be opened in both kolla and neutron since it is not determined what is causing this.

ENV: kolla stable/newton; containers are centos source.

--------- relevant kolla config settings --------------------
/etc/kolla/globals.yml
neutron_bridge_name: "br-vlan802,br-vlan805"
neutron_external_interface: "bond0.802,bond0.805"

#no dvr, lbaas, qos, etc
-------------------------------------------------------------

config files and logs can be found here
http://paste.openstack.org/show/589464/
http://paste.openstack.org/show/589286/
http://paste.openstack.org/show/589276/

snipplet from ovs log. 805 and 802 are the external bridges. They get disconnected as soon as they connect. br-tun and br-int does not show this behavior.
=> openvswitch/ovs-vswitchd.log <==
2016-11-15T12:15:51.586Z|01155|rconn|INFO|br-vlan805<->tcp:127.0.0.1:6633: connection closed by peer
2016-11-15T12:15:52.585Z|01156|rconn|INFO|br-vlan805<->tcp:127.0.0.1:6633: connecting...
2016-11-15T12:15:52.586Z|01157|rconn|INFO|br-vlan805<->tcp:127.0.0.1:6633: connected
2016-11-15T12:15:52.587Z|01158|rconn|INFO|br-vlan802<->tcp:127.0.0.1:6633: connection closed by peer
2016-11-15T12:15:53.584Z|01159|rconn|INFO|br-vlan802<->tcp:127.0.0.1:6633: connecting...
2016-11-15T12:15:53.585Z|01160|rconn|INFO|br-vlan802<->tcp:127.0.0.1:6633: connected
2016-11-15T12:15:53.586Z|01161|rconn|INFO|br-vlan805<->tcp:127.0.0.1:6633: connection closed by peer
2016-11-15T12:15:54.585Z|01162|rconn|INFO|br-vlan805<->tcp:127.0.0.1:6633: connecting...
2016-11-15T12:15:54.586Z|01163|rconn|INFO|br-vlan805<->tcp:127.0.0.1:6633: connected
2016-11-15T12:15:54.587Z|01164|rconn|INFO|br-vlan802<->tcp:127.0.0.1:6633: connection closed by peer
2016-11-15T12:15:55.584Z|01165|rconn|INFO|br-vlan802<->tcp:127.0.0.1:6633: connecting...
2016-11-15T12:15:55.586Z|01166|rconn|INFO|br-vlan802<->tcp:127.0.0.1:6633: connected
2016-11-15T12:15:55.587Z|01167|rconn|INFO|br-vlan805<->tcp:127.0.0.1:6633: connection closed by peer
2016-11-15T12:15:56.584Z|01168|rconn|INFO|br-vlan805<->tcp:127.0.0.1:6633: connecting...
2016-11-15T12:15:56.586Z|01169|rconn|INFO|br-vlan805<->tcp:127.0.0.1:6633: connected
2016-11-15T12:15:56.587Z|01170|rconn|INFO|br-vlan802<->tcp:127.0.0.1:6633: connection closed by peer
2016-11-15T12:15:57.585Z|01171|rconn|INFO|br-vlan802<->tcp:127.0.0.1:6633: connecting...
2016-11-15T12:15:57.586Z|01172|rconn|INFO|br-vlan802<->tcp:127.0.0.1:6633: connected
2016-11-15T12:15:57.586Z|01173|rconn|INFO|br-vlan805<->tcp:127.0.0.1:6633: connection closed by peer
2016-11-15T12:15:58.585Z|01174|rconn|INFO|br-vlan805<->tcp:127.0.0.1:6633: connecting...
2016-11-15T12:15:58.586Z|01175|rconn|INFO|br-vlan805<->tcp:127.0.0.1:6633: connected
2016-11-15T12:15:58.587Z|01176|rconn|INFO|br-vlan802<->tcp:127.0.0.1:6633: connection closed by peer
2016-11-15T12:15:59.585Z|01177|rconn|INFO|br-vlan802<->tcp:127.0.0.1:6633: connecting...
2016-11-15T12:15:59.586Z|01178|rconn|INFO|br-vlan802<->tcp:127.0.0.1:6633: connected
2016-11-15T12:15:59.587Z|01179|rconn|INFO|br-vlan805<->tcp:127.0.0.1:6633: connection closed by peer
2016-11-15T12:16:00.585Z|01180|rconn|INFO|br-vlan805<->tcp:127.0.0.1:6633: connecting...
2016-11-15T12:16:00.587Z|01181|rconn|INFO|br-vlan805<->tcp:127.0.0.1:6633: connected
2016-11-15T12:16:00.587Z|01182|rconn|INFO|br-vlan802<->tcp:127.0.0.1:6633: connection closed by peer
2016-11-15T12:16:01.584Z|01183|rconn|INFO|br-vlan802<->tcp:127.0.0.1:6633: connecting...
2016-11-15T12:16:01.585Z|01184|rconn|INFO|br-vlan802<->tcp:127.0.0.1:6633: connected
2016-11-15T12:16:01.586Z|01185|rconn|INFO|br-vlan805<->tcp:127.0.0.1:6633: connection closed by peer
2016-11-15T12:16:02.585Z|01186|rconn|INFO|br-vlan805<->tcp:127.0.0.1:6633: connecting...

[Paul Bourke (pauldbourke)]

I've tested this and am unable to reproduce using the following configurations:

* base: oraclelinux, type: source, kolla: master
* base: centos, type: source, kolla: newton

In both cases I'm using libvirt guests with kvm to host my Kolla cluster, with virtio interfaces attached to a libvirt bridge for networking. I'll try post as much logs and output as I can so we can try and compare and see what might be wrong in your environment.

The following info is from my most recent test which is the centos newton case.

/etc/kolla/globals.yml: http://paste.fedoraproject.org/484170/46596114/
/etc/kolla/kolla-build.conf: http://paste.fedoraproject.org/484171/79465984/
inventory: http://paste.fedoraproject.org/484173/47946599/

Commands run on the compute node:

'ip a' - http://paste.fedoraproject.org/484174/94661101/
'docker exec openvswitch_vswitchd ovs-vsctl show' http://paste.fedoraproject.org/484175/46613714/
/etc/kolla/nova-compute/nova.conf: http://paste.fedoraproject.org/484176/66179147
/etc/kolla/neutron-openvswitch-agent/ml2_conf.ini: http://paste.fedoraproject.org/484177/46623014/
/var/lib/docker/volumes/kolla_logs/_data/openvswitch/ovs-vswitchd.log: http://paste.fedoraproject.org/484178/46625914/

[Paul Bourke (pauldbourke)]

Also, so we're sure we're actually following a similar use case, what I did to actually test was to update tools/init-runonce for my external network:

# git diff tools/init-runonce
diff --git a/tools/init-runonce b/tools/init-runonce
index 7fb50f7..248cdc2 100755
--- a/tools/init-runonce
+++ b/tools/init-runonce
@@ -56,7 +56,8 @@ if [[ "${NETWORK_MANAGER}" == "nova" ]] ; then
 else
     echo Configuring neutron.
     neutron net-create public1 --router:external --provider:physical_network physnet1 --provider:network_type flat
- neutron subnet-create --name 1-subnet --disable-dhcp --allocation-pool start=10.0.2.150,end=10.0.2.199 public1 10.0.2.0/24 --gateway 10.0.2.1
+ neutron subnet-create --name 1-subnet --disable-dhcp --allocation-pool start=192.168.6.100,end=192.168.6.150 public1 192.168.6.0/24 --gateway 192.168.6.1
+
     neutron net-create demo-net --provider:network_type vxlan
     neutron subnet-create demo-net 10.0.0.0/24 --name demo-subnet --gateway 10.0.0.1 --dns-nameservers list=true 8.8.8.8
     neutron router-create demo-router

Then:

tools/init-runonce
openstack server create --image cirros --flavor m1.tiny --key-name mykey --nic net-id=cb8ae6de-bf97-4a23-a009-bf6b7d173e0b demo1
openstack floating ip create public1
openstack server add floating ip demo1 192.168.6.109
# ping 192.168.6.109
PING 192.168.6.109 (192.168.6.109) 56(84) bytes of data.
64 bytes from 192.168.6.109: icmp_seq=1 ttl=63 time=7.23 ms
64 bytes from 192.168.6.109: icmp_seq=2 ttl=63 time=12.9 ms
^C

[Paul Bourke (pauldbourke)]

docker exec openvswitch_vswitchd ovs-ofctl dump-flows br-int [http://paste.fedoraproject.org/484215/94713211/]

docker exec openvswitch_vswitchd ovs-ofctl dump-flows br-ex1 [http://paste.fedoraproject.org/484216/94713531]

docker exec openvswitch_vswitchd ovs-ofctl dump-flows br-ex2 [http://paste.fedoraproject.org/484215/94713211/]

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for kolla because there has been no activity for 60 days.]