network-ovn with devstack deployed failed because of no bridge br-int/br-tun/br-ex

[yaowei]

Deploying devstack with networking-ovn following with the doc http://docs.openstack.org/developer/networking-ovn/testing.html , error occured in running stack.sh.

Traceback below here:
2016-03-14 09:40:16.002 | cd datapath/linux && make modules_install
2016-03-14 09:40:16.007 | make[1]: Entering directory `/opt/stack/ovs/datapath/linux'
2016-03-14 09:40:16.008 | make -C /lib/modules/3.10.0-327.10.1.el7.x86_64/build M=/opt/stack/ovs/datapath/linux modules_install
2016-03-14 09:40:16.343 | make[2]: Entering directory `/usr/src/kernels/3.10.0-327.10.1.el7.x86_64'
2016-03-14 09:40:16.362 | INSTALL /opt/stack/ovs/datapath/linux/openvswitch.ko
2016-03-14 09:40:16.470 | Can't read private key
2016-03-14 09:40:16.472 | INSTALL /opt/stack/ovs/datapath/linux/vport-geneve.ko
2016-03-14 09:40:16.500 | Can't read private key
2016-03-14 09:40:16.502 | INSTALL /opt/stack/ovs/datapath/linux/vport-gre.ko
2016-03-14 09:40:16.529 | Can't read private key
2016-03-14 09:40:16.531 | INSTALL /opt/stack/ovs/datapath/linux/vport-lisp.ko
2016-03-14 09:40:16.560 | Can't read private key
2016-03-14 09:40:16.562 | INSTALL /opt/stack/ovs/datapath/linux/vport-stt.ko
2016-03-14 09:40:16.592 | Can't read private key
2016-03-14 09:40:16.594 | INSTALL /opt/stack/ovs/datapath/linux/vport-vxlan.ko
2016-03-14 09:40:16.621 | Can't read private key
2016-03-14 09:40:16.635 | DEPMOD 3.10.0-327.10.1.el7.x86_64
2016-03-14 09:40:20.978 | make[2]: Leaving directory `/usr/src/kernels/3.10.0-327.10.1.el7.x86_64'
2016-03-14 09:40:20.978 | depmod `sed -n 's/#define UTS_RELEASE "\([^"]*\)"/\1/p' /lib/modules/3.10.0-327.10.1.el7.x86_64/build/include/generated/utsrelease.h`
2016-03-14 09:40:21.313 | make[1]: Leaving directory `/opt/stack/ovs/datapath/linux'
2016-03-14 09:40:21.333 | modprobe: FATAL: Module openvswitch is in use.
2016-03-14 09:40:21.335 | Error on exit
2016-03-14 09:40:21.831 | ovs-vsctl: unix:/usr/local/var/run/openvswitch/db.sock: database connection failed (No such file or directory)
2016-03-14 09:40:21.842 | ovs-ofctl: br-int is not a bridge or a socket
2016-03-14 09:40:21.852 | ovs-ofctl: br-tun is not a bridge or a socket
2016-03-14 09:40:21.862 | ovs-ofctl: br-ex is not a bridge or a socket
2016-03-14 09:40:21.873 | ovs-ofctl: br-int is not a bridge or a socket
2016-03-14 09:40:21.884 | ovs-ofctl: br-tun is not a bridge or a socket
2016-03-14 09:40:21.894 | ovs-ofctl: br-ex is not a bridge or a socket

 In this process , stack.sh will install openvswitch when processing neutron, then it will uninstall openvswitch and make && make install openvswitch from ovs code source again.

 Because of uninstalling of openvswitch, br-int and other brigeds loss from ovs, so the following networking-ovn process come accross the error that no bridges br-int/br-ex/br-tun.

Environment:
CentOs 7.1 & kernel version 3.10.0-327.10.1.el7.x86_64
local.conf of devstack is just a copy of networking-ovn/examples, and it is below here:

#
# Sample DevStack local.conf.
#
# This sample file is intended to be used for your typical DevStack environment
# that's running all of OpenStack on a single host. This can also be used as
# the first host of a multi-host test environment.
#
# No changes to this sample configuration are required for this to work.
#

[[local|localrc]]

HOST_IP=127.0.0.1
# OFFLINE=True

LOGFILE=/opt/stack/logs/stack.log
LOG_COLOR=True
SCREEN_LOGDIR=/opt/stack/logs

DATABASE_PASSWORD=password
RABBIT_PASSWORD=password
SERVICE_PASSWORD=password
SERVICE_TOKEN=password
ADMIN_PASSWORD=password

disable_service tempest
enable_service horizon
# The DevStack plugin defaults to using the ovn branch from the official ovs
# repo. You can optionally use a different one. For example, you may want to
# use the latest patches in blp's ovn branch:
#OVN_REPO=http://github.com/blp/ovs-reviews.git
#OVN_BRANCH=origin/ovn

enable_plugin networking-ovn https://github.com/openstack/networking-ovn
enable_service ovn-northd
enable_service ovn-controller

# Use Neutron instead of nova-network
disable_service n-net
enable_service q-svc

# OVN currently uses the conventional DHCP and metadata agents. For
# single-node deployments, enable them on the controller node. For
# multi-node deployments, consider disabling them on the controller
# node and enabling them on the compute node(s) to provide a more
# realistic environment. For more information, see
# computenode-local.conf.sample.
enable_service q-dhcp
enable_service q-meta

# We have to disable the neutron L2 agent. OVN does not use the L2 agent.
disable_service q-agt

# How to connect to ovsdb-server hosting the OVN databases.
#OVN_REMOTE=tcp:$SERVICE_HOST:6640

# A UUID to uniquely identify this system. If one is not specified, a random
# one will be generated and saved in the file 'ovn-uuid' for re-use in future
# DevStack runs.
#OVN_UUID=

# Whether to enable using OVN's L3 functionality. If this value is disabled,
# OpenStack will use the q-l3 functionality. If you set OVN_L3_MODE to False,
# you must also enable the q-l3 service.
# By default OVN_L3_MODE is True
#OVN_L3_MODE=False
#enable_service q-l3
disable_service q-l3

# Whether or not to build custom openvswitch kernel modules from the ovs git
# tree. This is enabled by default. This is required unless your distro kernel
# includes ovs+conntrack support. This support was first released in Linux 4.3,
# and will likely be backported by some distros.
#OVN_BUILD_MODULES=True

# If you enable the DHCP agent, configure it to provide a route for
# metadata to instances.
ENABLE_ISOLATED_METADATA=True

# Enable two DHCP agents per neutron subnet with support for availability
# zones. Requires a multi-node deployment.
#[[post-config|/$NEUTRON_CONF]]
#[DEFAULT]
#network_scheduler_driver = neutron.scheduler.dhcp_agent_scheduler.AZAwareWeightScheduler
#dhcp_load_type = networks
#dhcp_agents_per_network = 2

# If you enable the DHCP agent, you can configure the availability
# zone name (default is nova).
#[[post-config|$Q_DHCP_CONF_FILE]]
#[AGENT]
#availability_zone = nova

probable fix solutions:
1. provent install openvswitch and leave it installed when processing network-ovn
2. start openvswitch
3. create bridges br-int/be-tun/br-ex
4. ovs-ofctl ***

[Russell Bryant (russellb)]

Thanks for the report!

I know this is working successfully on Fedora 23 and Ubuntu 14.04, but it has been a whlie since I've tried it on CentOS. This could be CentOS specific. I'll try to reproduce. Was anything running on this system before you ran devstack, or is this a fresh environment?

In any case, preventing the ovs package from being installed and then immediately uninstalled again would be an improvement.

[yaowei (yaowei-deactivatedaccount)]

It's a fresh centos7.1 machine before I run stack.sh.

This bug https://bugs.launchpad.net/networking-ovn/+bug/1537981 has also mentioned about preventing install openvswitch but I don't know how to deal with it.

[Richard Theis (rtheis)]

I've noticed that br-int no longer exists on my compute node after it is restarted. I'm using Ubuntu 14.04.

[Anil Shashikumar Belur (askb23)]

I am hitting the same issue with devstack, on Ubuntu 14.04. Is there are work around for this ??

World dumping... see /opt/stack/logs/worlddump-2016-03-25-042359.txt for details
ovs-ofctl: br-tun is not a bridge or a socket
ovs-ofctl: br-tun is not a bridge or a socket
==============================================

Here is my orginal local.conf:

==========================================================================
My local.conf:
[[local|localrc]]
# Default passwords
ADMIN_PASSWORD=pass12
MYSQL_PASSWORD=pass12
RABBIT_PASSWORD=pass12
SERVICE_PASSWORD=pass12
SERVICE_TOKEN=pass12
public_interface="eth2"

SCREEN_LOGDIR=/opt/stack/logs
HOST_IP=192.168.27.100
FLOATING_RANGE=172.24.4.0/24
 PUBLIC_INTERFACE=$public_interface
enable_plugin magnum https://git.openstack.org/openstack/magnum master

enable_plugin barbican https://git.openstack.org/openstack/barbican master
VOLUME_BACKING_FILE_SIZE=20G

disable_service n-net
enable_service q-svc
enable_service q-agt
enable_service q-dhcp
enable_service q-l3
enable_service q-meta
enable_service neutron
enable_service heat
enable_service h-api
enable_service h-api-cfn
enable_service h-api-cw
enable_service h-eng

# Disable security groups entirely
Q_USE_SECGROUP=False
LIBVIRT_FIREWALL_DRIVER=nova.virt.firewall.NoopFirewallDriver

disable_service tempest

[Richard Theis (rtheis)]

Hi Anil, your local.conf should not enable q-agt. Nor should it enable q-l3 without disabling OVN native L3. Please follow the devstack deployment instructions here: http://docs.openstack.org/developer/networking-ovn/testing.html

[Anil Shashikumar Belur (askb23)]

Possible workaround: This looks like, the issue is with 'unstack.sh' - which does not clean up the ovs / network setup when run. Therefore its not required to re-create your VM from the start again, instead just undo/del all the virtual networking setup {br-int, br-eth1, br-tun} before running `stack.sh` again.

for instance:
ovs-vsctl del-br br-int
ovs-vsctl del-br br-eth1

#ip add
ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:b6:ee:51 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 brd 10.0.2.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:feb6:ee51/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:92:b3:8f brd ff:ff:ff:ff:ff:ff
    inet 192.168.27.100/24 brd 192.168.27.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe92:b38f/64 scope link
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 08:00:27:5a:8b:41 brd ff:ff:ff:ff:ff:ff
5: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 86:7a:26:7a:1b:fd brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
6: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default
    link/ether 1e:8c:3a:e8:06:7e brd ff:ff:ff:ff:ff:ff
7: br-int: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
    link/ether 66:0c:99:93:c7:48 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::e03e:97ff:fe76:f5c6/64 scope link
       valid_lft forever preferred_lft forever
8: br-ex: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
    link/ether ea:9a:1d:8b:a0:4d brd ff:ff:ff:ff:ff:ff
    inet 172.24.4.1/24 scope global br-ex
       valid_lft forever preferred_lft forever
    inet6 2001:db8::2/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::3042:50ff:fe66:3172/64 scope link
       valid_lft forever preferred_lft forever
9: br-tun: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
    link/ether a6:c2:33:3b:ff:46 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::40d6:2ff:fed5:a68f/64 scope link
       valid_lft forever preferred_lft forever

[Richard Theis (rtheis)]

Hi Anil, can you please retry using the latest code. I was successfully able to use unstack.sh before a reboot and stack.sh after a reboot.

[yaowei (yaowei-deactivatedaccount)]

This problem doesn't occur now