DevStack install with Neutron fails with "RTNETLINK answers: Network is unreachable"

[Mike Spreitzer]

I tested http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-a-single-interface with the stable/liberty branch, and stack.sh failed.

I used exactly the configuration described in that doc section, with only a link-local IPv6 address on the one network interface, but with a couple of changes to local.conf. One is that I added "IP_VERSION=4" to avoid any attempt to configure Neutron to use IPv6. The other is that I added a few lines to improve the logging --- mainly making the logs go to files (which I will attach to this bug).

I used Ubuntu 14.04.3, fully updated. That means kernel 3.19.0-42-generic.

I downloaded openvswitch-2.4.0.tar.gz and built the Debian packages for OpenVswitch according to the instruction contained therein. For OVS installation: I eschewed the kernel module I just built, instead relying on what is shipped with Linux 3.19, and I installed the openvswitch-switch and openvswitch-common debian packages I had just built.

When I ran stack.sh, it aborted with problems configuring br-ex. In the attached archive, under home/mspreitz/devstack/, you will find a file named typescript, produced by `script`, which shows some of the state before I started stack.sh (as well as a typescript of running stack.sh). You will also find postmortem.log, showing some of the state right after stack.sh aborted. Additionally there is net-recovery.log, showing how I recovered to a working network configuration --- hopefully this gives further clues to what went wrong in stack.sh. You will see that what I did is to change the state of the br-ex link to UP and then add the appropriate route through br-ex (although at least part of that command was wrong and/or unnecessary). You will also find that opt/stack/logs/ has all the logs that stack.sh normally produces.

I suspect the problem centers around the br-ex link not being "up". But that's only a slightly informed guess.

For easy reference, here is the end of stack.sh.log:

2015-12-30 06:56:15.111 | + IP_BRD='172.18.161.6/24 brd 172.18.161.255'
2015-12-30 06:56:15.111 | + [[ inet == \i\n\e\t\6 ]]
2015-12-30 06:56:15.111 | + '[' 172.18.161.1 '!=' '' ']'
2015-12-30 06:56:15.111 | + ADD_DEFAULT_ROUTE='sudo ip r replace default via 172.18.161.1 dev br-ex'
2015-12-30 06:56:15.111 | + [[ True == \T\r\u\e ]]
2015-12-30 06:56:15.111 | + ADD_OVS_PORT='sudo ovs-vsctl --may-exist add-port br-ex eth0'
2015-12-30 06:56:15.111 | + [[ 172.18.161.6/24 brd 172.18.161.255 != '' ]]
2015-12-30 06:56:15.111 | + IP_DEL='sudo ip addr del 172.18.161.6/24 brd 172.18.161.255 dev eth0'
2015-12-30 06:56:15.111 | + IP_ADD='sudo ip addr add 172.18.161.6/24 brd 172.18.161.255 dev br-ex'
2015-12-30 06:56:15.111 | + sudo ip addr del 172.18.161.6/24 brd 172.18.161.255 dev eth0
2015-12-30 06:56:15.120 | + sudo ip addr add 172.18.161.6/24 brd 172.18.161.255 dev br-ex
2015-12-30 06:56:15.130 | + sudo ovs-vsctl --may-exist add-port br-ex eth0
2015-12-30 06:56:15.173 | + sudo ip r replace default via 172.18.161.1 dev br-ex
2015-12-30 06:56:15.184 | RTNETLINK answers: Network is unreachable
2015-12-30 06:56:15.186 | + exit_trap
2015-12-30 06:56:15.186 | + local r=2
2015-12-30 06:56:15.186 | ++ jobs -p
2015-12-30 06:56:15.187 | + jobs=
2015-12-30 06:56:15.188 | + [[ -n '' ]]
2015-12-30 06:56:15.188 | + kill_spinner
2015-12-30 06:56:15.188 | + '[' '!' -z '' ']'
2015-12-30 06:56:15.188 | + [[ 2 -ne 0 ]]
2015-12-30 06:56:15.188 | + echo 'Error on exit'
2015-12-30 06:56:15.188 | Error on exit
2015-12-30 06:56:15.188 | + [[ -z /opt/stack/logs ]]
2015-12-30 06:56:15.188 | + /home/mspreitz/devstack/tools/worlddump.py -d /opt/stack/logs
2015-12-30 06:56:15.549 | + exit 2

[Mike Spreitzer (mike-spreitzer)]

Also, perhaps this is relevant: https://bugs.launchpad.net/openstack-manuals/+bug/1529444/comments/11

[Mike Spreitzer (mike-spreitzer)]

Following are a few more observations, taken after net-recovery.log but no other changes.

mspreitz@devstack-1:~$ brctl show
bridge name bridge id STP enabled interfaces
virbr0 8000.000000000000 yes
mspreitz@devstack-1:~$
mspreitz@devstack-1:~$ sudo ovs-vsctl show
d1d737a2-54d0-4c30-9f73-2b318963be2a
    Bridge br-int
        Port br-int
            Interface br-int
                type: internal
    Bridge br-ex
        Port "eth0"
            Interface "eth0"
        Port br-ex
            Interface br-ex
                type: internal
    ovs_version: "2.4.0"
mspreitz@devstack-1:~$
mspreitz@devstack-1:~$
mspreitz@devstack-1:~$ sudo ovs-ofctl dump-flows br-ex
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=44473.152s, table=0, n_packets=7814, n_bytes=5523423, idle_age=2, priority=0 actions=NORMAL
mspreitz@devstack-1:~$ sudo ovs-ofctl dump-flows br-int
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=44479.069s, table=0, n_packets=0, n_bytes=0, idle_age=44479, priority=0 actions=NORMAL
mspreitz@devstack-1:~$
mspreitz@devstack-1:~$
mspreitz@devstack-1:~$ sudo ovs-appctl bridge/dump-flows br-ex
duration=44493s, n_packets=7969, n_bytes=5537101, priority=0,actions=NORMAL
table_id=254, duration=44493s, n_packets=0, n_bytes=0, priority=2,recirc_id=0,actions=drop
table_id=254, duration=44493s, n_packets=0, n_bytes=0, priority=0,reg0=0x1,actions=controller(reason=no_match)
table_id=254, duration=44493s, n_packets=0, n_bytes=0, priority=0,reg0=0x2,actions=drop
table_id=254, duration=44493s, n_packets=0, n_bytes=0, priority=0,reg0=0x3,actions=drop
mspreitz@devstack-1:~$
mspreitz@devstack-1:~$ sudo ovs-appctl bridge/dump-flows br-int
duration=44500s, n_packets=0, n_bytes=0, priority=0,actions=NORMAL
table_id=254, duration=44500s, n_packets=0, n_bytes=0, priority=2,recirc_id=0,actions=drop
table_id=254, duration=44500s, n_packets=0, n_bytes=0, priority=0,reg0=0x1,actions=controller(reason=no_match)
table_id=254, duration=44500s, n_packets=0, n_bytes=0, priority=0,reg0=0x2,actions=drop
table_id=254, duration=44500s, n_packets=0, n_bytes=0, priority=0,reg0=0x3,actions=drop
mspreitz@devstack-1:~$
mspreitz@devstack-1:~$
mspreitz@devstack-1:~$ sudo ovs-dpctl dump-flows
recirc_id(0),in_port(2),eth(src=08:00:27:5c:d2:77,dst=08:00:27:0d:51:c9),eth_type(0x0800),ipv4(frag=no), packets:243, bytes:28128, used:0.012s, flags:P., actions:3
recirc_id(0),in_port(3),eth(src=08:00:27:0d:51:c9,dst=08:00:27:5c:d2:77),eth_type(0x0800),ipv4(frag=no), packets:410, bytes:35260, used:0.013s, flags:P., actions:2
mspreitz@devstack-1:~$
mspreitz@devstack-1:~$
mspreitz@devstack-1:~$ sudo ovs-appctl dpif/dump-flows br-ex
recirc_id(0),in_port(2),eth(src=08:00:27:5c:d2:77,dst=08:00:27:0d:51:c9),eth_type(0x0800),ipv4(frag=no), packets:39, bytes:3730, used:0.012s, flags:P., actions:3
recirc_id(0),in_port(3),eth(src=08:00:27:0d:51:c9,dst=08:00:27:5c:d2:77),eth_type(0x0800),ipv4(frag=no), packets:79, bytes:6306, used:0.012s, flags:P., actions:2
mspreitz@devstack-1:~$ sudo ovs-appctl dpif/dump-flows br-int
mspreitz@devstack-1:~$

[Mike Spreitzer (mike-spreitzer)]

BTW, on a clone of the machine I used to build the Debian packages for OVS 2.4.0, I tried to install the kernel module from OVS. I followed the instructions to install the openvswitch-datapath-dkms Debian package I just built. You can see the typescript at https://bugs.launchpad.net/devstack/+bug/1530062

It is not clear to me whether this went as expected. When I look at `lsmod` output I do not see anything that is obviously for OVS.

[Sean M. Collins (scollins)]

So this is after you built your own version of Open vSwitch from source? I don't think this is a DevStack issue.

[Mike Spreitzer (mike-spreitzer)]

What's my alternative if I'm using Ubuntu?

[Mike Spreitzer (mike-spreitzer)]

DevStack is known to not work with versions 15.04 or 15.10 of Ubuntu. The standard sources for Ubuntu 14.04 include only OVS version 2.0.2. The discussion at https://bugs.launchpad.net/openstack-manuals/+bug/1529444 seems to say that I need at least version 2.1. Is there some standard source for Debian packages for version 2.1 or higher of OVS that install on Ubuntu 14.04?

Also: is the 3.19 kernel's support for OVS good enough or do I have to install a kernel module for OVS (I ask because of the discussion at https://github.com/openvswitch/ovs/blob/master/FAQ.md#q-are-all-features-available-with-all-datapaths)?

[Mike Spreitzer (mike-spreitzer)]

Thanks, Han, for pointing out that this has been fixed in HEAD. Can we back-port it to stable/liberty?

[Mike Spreitzer (mike-spreitzer)]

Fix proposed to stable/liberty: https://review.openstack.org/#/c/263065/