Activity log for bug #1530062

Date Who What changed Old value New value Message
2015-12-30 08:52:01 Mike Spreitzer bug added bug
2015-12-30 08:52:01 Mike Spreitzer attachment added 1nic-ovs-upstream-fail.zip https://bugs.launchpad.net/bugs/1530062/+attachment/4542077/+files/1nic-ovs-upstream-fail.zip
2015-12-30 08:53:53 Mike Spreitzer description I tested http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-a-single-interface and stack.sh failed. I used exactly the configuration described in that doc section, with only a link-local IPv6 address on the one network interface, but with a couple of changes to local.conf. One is that I added "IP_VERSION=4" to avoid any attempt to configure Neutron to use IPv6. The other is that I added a few lines to improve the logging --- mainly making the logs go to files (which I will attach to this bug). I used Ubuntu 14.04.3, fully updated. That means kernel 3.19.0-42-generic. I downloaded openvswitch-2.4.0.tar.gz and built the Debian packages for OpenVswitch according to the instruction contained therein. For OVS installation: I eschewed the kernel module I just built, instead relying on what is shipped with Linux 3.19, and I installed the openvswitch-switch and openvswitch-common debian packages I had just built. When I ran stack.sh, it aborted with problems configuring br-ex. In the attached archive, under home/mspreitz/devstack/, you will find a file named typescript, produced by `script`, which shows some of the state before I started stack.sh (as well as a typescript of running stack.sh). You will also find postmortem.log, showing some of the state right after stack.sh aborted. Additionally there is net-recovery.log, showing how I recovered to a working network configuration --- hopefully this gives further clues to what went wrong in stack.sh. You will see that what I did is to change the state of the br-ex link to UP and then add the appropriate route through br-ex (although at least part of that command was unnecessary). You will also find that opt/stack/logs/ has all the logs that stack.sh normally produces. I suspect the problem centers around the br-ex link not being "up". But that's only a slightly informed guess. For easy reference, here is the end of stack.sh.log: 2015-12-30 06:56:15.111 | + IP_BRD='172.18.161.6/24 brd 172.18.161.255' 2015-12-30 06:56:15.111 | + [[ inet == \i\n\e\t\6 ]] 2015-12-30 06:56:15.111 | + '[' 172.18.161.1 '!=' '' ']' 2015-12-30 06:56:15.111 | + ADD_DEFAULT_ROUTE='sudo ip r replace default via 172.18.161.1 dev br-ex' 2015-12-30 06:56:15.111 | + [[ True == \T\r\u\e ]] 2015-12-30 06:56:15.111 | + ADD_OVS_PORT='sudo ovs-vsctl --may-exist add-port br-ex eth0' 2015-12-30 06:56:15.111 | + [[ 172.18.161.6/24 brd 172.18.161.255 != '' ]] 2015-12-30 06:56:15.111 | + IP_DEL='sudo ip addr del 172.18.161.6/24 brd 172.18.161.255 dev eth0' 2015-12-30 06:56:15.111 | + IP_ADD='sudo ip addr add 172.18.161.6/24 brd 172.18.161.255 dev br-ex' 2015-12-30 06:56:15.111 | + sudo ip addr del 172.18.161.6/24 brd 172.18.161.255 dev eth0 2015-12-30 06:56:15.120 | + sudo ip addr add 172.18.161.6/24 brd 172.18.161.255 dev br-ex 2015-12-30 06:56:15.130 | + sudo ovs-vsctl --may-exist add-port br-ex eth0 2015-12-30 06:56:15.173 | + sudo ip r replace default via 172.18.161.1 dev br-ex 2015-12-30 06:56:15.184 | RTNETLINK answers: Network is unreachable 2015-12-30 06:56:15.186 | + exit_trap 2015-12-30 06:56:15.186 | + local r=2 2015-12-30 06:56:15.186 | ++ jobs -p 2015-12-30 06:56:15.187 | + jobs= 2015-12-30 06:56:15.188 | + [[ -n '' ]] 2015-12-30 06:56:15.188 | + kill_spinner 2015-12-30 06:56:15.188 | + '[' '!' -z '' ']' 2015-12-30 06:56:15.188 | + [[ 2 -ne 0 ]] 2015-12-30 06:56:15.188 | + echo 'Error on exit' 2015-12-30 06:56:15.188 | Error on exit 2015-12-30 06:56:15.188 | + [[ -z /opt/stack/logs ]] 2015-12-30 06:56:15.188 | + /home/mspreitz/devstack/tools/worlddump.py -d /opt/stack/logs 2015-12-30 06:56:15.549 | + exit 2 I tested http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-a-single-interface and stack.sh failed. I used exactly the configuration described in that doc section, with only a link-local IPv6 address on the one network interface, but with a couple of changes to local.conf. One is that I added "IP_VERSION=4" to avoid any attempt to configure Neutron to use IPv6. The other is that I added a few lines to improve the logging --- mainly making the logs go to files (which I will attach to this bug). I used Ubuntu 14.04.3, fully updated. That means kernel 3.19.0-42-generic. I downloaded openvswitch-2.4.0.tar.gz and built the Debian packages for OpenVswitch according to the instruction contained therein. For OVS installation: I eschewed the kernel module I just built, instead relying on what is shipped with Linux 3.19, and I installed the openvswitch-switch and openvswitch-common debian packages I had just built. When I ran stack.sh, it aborted with problems configuring br-ex. In the attached archive, under home/mspreitz/devstack/, you will find a file named typescript, produced by `script`, which shows some of the state before I started stack.sh (as well as a typescript of running stack.sh). You will also find postmortem.log, showing some of the state right after stack.sh aborted. Additionally there is net-recovery.log, showing how I recovered to a working network configuration --- hopefully this gives further clues to what went wrong in stack.sh. You will see that what I did is to change the state of the br-ex link to UP and then add the appropriate route through br-ex (although at least part of that command was wrong and/or unnecessary). You will also find that opt/stack/logs/ has all the logs that stack.sh normally produces. I suspect the problem centers around the br-ex link not being "up". But that's only a slightly informed guess. For easy reference, here is the end of stack.sh.log: 2015-12-30 06:56:15.111 | + IP_BRD='172.18.161.6/24 brd 172.18.161.255' 2015-12-30 06:56:15.111 | + [[ inet == \i\n\e\t\6 ]] 2015-12-30 06:56:15.111 | + '[' 172.18.161.1 '!=' '' ']' 2015-12-30 06:56:15.111 | + ADD_DEFAULT_ROUTE='sudo ip r replace default via 172.18.161.1 dev br-ex' 2015-12-30 06:56:15.111 | + [[ True == \T\r\u\e ]] 2015-12-30 06:56:15.111 | + ADD_OVS_PORT='sudo ovs-vsctl --may-exist add-port br-ex eth0' 2015-12-30 06:56:15.111 | + [[ 172.18.161.6/24 brd 172.18.161.255 != '' ]] 2015-12-30 06:56:15.111 | + IP_DEL='sudo ip addr del 172.18.161.6/24 brd 172.18.161.255 dev eth0' 2015-12-30 06:56:15.111 | + IP_ADD='sudo ip addr add 172.18.161.6/24 brd 172.18.161.255 dev br-ex' 2015-12-30 06:56:15.111 | + sudo ip addr del 172.18.161.6/24 brd 172.18.161.255 dev eth0 2015-12-30 06:56:15.120 | + sudo ip addr add 172.18.161.6/24 brd 172.18.161.255 dev br-ex 2015-12-30 06:56:15.130 | + sudo ovs-vsctl --may-exist add-port br-ex eth0 2015-12-30 06:56:15.173 | + sudo ip r replace default via 172.18.161.1 dev br-ex 2015-12-30 06:56:15.184 | RTNETLINK answers: Network is unreachable 2015-12-30 06:56:15.186 | + exit_trap 2015-12-30 06:56:15.186 | + local r=2 2015-12-30 06:56:15.186 | ++ jobs -p 2015-12-30 06:56:15.187 | + jobs= 2015-12-30 06:56:15.188 | + [[ -n '' ]] 2015-12-30 06:56:15.188 | + kill_spinner 2015-12-30 06:56:15.188 | + '[' '!' -z '' ']' 2015-12-30 06:56:15.188 | + [[ 2 -ne 0 ]] 2015-12-30 06:56:15.188 | + echo 'Error on exit' 2015-12-30 06:56:15.188 | Error on exit 2015-12-30 06:56:15.188 | + [[ -z /opt/stack/logs ]] 2015-12-30 06:56:15.188 | + /home/mspreitz/devstack/tools/worlddump.py -d /opt/stack/logs 2015-12-30 06:56:15.549 | + exit 2
2015-12-30 08:54:50 Mike Spreitzer summary DevStack install fails with "RTNETLINK answers: Network is unreachable" DevStack install with Neutron fails with "RTNETLINK answers: Network is unreachable"
2015-12-30 08:56:20 Mike Spreitzer bug added subscriber Sean M. Collins
2015-12-31 01:24:14 Sean M. Collins devstack: status New Incomplete
2016-01-01 10:14:26 Han Manjong devstack: assignee Han Manjong (aksmj8855)
2016-01-01 13:50:44 Han Manjong devstack: assignee Han Manjong (aksmj8855)
2016-01-01 13:51:12 Han Manjong marked as duplicate 1469596
2016-01-02 18:56:05 Mike Spreitzer description I tested http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-a-single-interface and stack.sh failed. I used exactly the configuration described in that doc section, with only a link-local IPv6 address on the one network interface, but with a couple of changes to local.conf. One is that I added "IP_VERSION=4" to avoid any attempt to configure Neutron to use IPv6. The other is that I added a few lines to improve the logging --- mainly making the logs go to files (which I will attach to this bug). I used Ubuntu 14.04.3, fully updated. That means kernel 3.19.0-42-generic. I downloaded openvswitch-2.4.0.tar.gz and built the Debian packages for OpenVswitch according to the instruction contained therein. For OVS installation: I eschewed the kernel module I just built, instead relying on what is shipped with Linux 3.19, and I installed the openvswitch-switch and openvswitch-common debian packages I had just built. When I ran stack.sh, it aborted with problems configuring br-ex. In the attached archive, under home/mspreitz/devstack/, you will find a file named typescript, produced by `script`, which shows some of the state before I started stack.sh (as well as a typescript of running stack.sh). You will also find postmortem.log, showing some of the state right after stack.sh aborted. Additionally there is net-recovery.log, showing how I recovered to a working network configuration --- hopefully this gives further clues to what went wrong in stack.sh. You will see that what I did is to change the state of the br-ex link to UP and then add the appropriate route through br-ex (although at least part of that command was wrong and/or unnecessary). You will also find that opt/stack/logs/ has all the logs that stack.sh normally produces. I suspect the problem centers around the br-ex link not being "up". But that's only a slightly informed guess. For easy reference, here is the end of stack.sh.log: 2015-12-30 06:56:15.111 | + IP_BRD='172.18.161.6/24 brd 172.18.161.255' 2015-12-30 06:56:15.111 | + [[ inet == \i\n\e\t\6 ]] 2015-12-30 06:56:15.111 | + '[' 172.18.161.1 '!=' '' ']' 2015-12-30 06:56:15.111 | + ADD_DEFAULT_ROUTE='sudo ip r replace default via 172.18.161.1 dev br-ex' 2015-12-30 06:56:15.111 | + [[ True == \T\r\u\e ]] 2015-12-30 06:56:15.111 | + ADD_OVS_PORT='sudo ovs-vsctl --may-exist add-port br-ex eth0' 2015-12-30 06:56:15.111 | + [[ 172.18.161.6/24 brd 172.18.161.255 != '' ]] 2015-12-30 06:56:15.111 | + IP_DEL='sudo ip addr del 172.18.161.6/24 brd 172.18.161.255 dev eth0' 2015-12-30 06:56:15.111 | + IP_ADD='sudo ip addr add 172.18.161.6/24 brd 172.18.161.255 dev br-ex' 2015-12-30 06:56:15.111 | + sudo ip addr del 172.18.161.6/24 brd 172.18.161.255 dev eth0 2015-12-30 06:56:15.120 | + sudo ip addr add 172.18.161.6/24 brd 172.18.161.255 dev br-ex 2015-12-30 06:56:15.130 | + sudo ovs-vsctl --may-exist add-port br-ex eth0 2015-12-30 06:56:15.173 | + sudo ip r replace default via 172.18.161.1 dev br-ex 2015-12-30 06:56:15.184 | RTNETLINK answers: Network is unreachable 2015-12-30 06:56:15.186 | + exit_trap 2015-12-30 06:56:15.186 | + local r=2 2015-12-30 06:56:15.186 | ++ jobs -p 2015-12-30 06:56:15.187 | + jobs= 2015-12-30 06:56:15.188 | + [[ -n '' ]] 2015-12-30 06:56:15.188 | + kill_spinner 2015-12-30 06:56:15.188 | + '[' '!' -z '' ']' 2015-12-30 06:56:15.188 | + [[ 2 -ne 0 ]] 2015-12-30 06:56:15.188 | + echo 'Error on exit' 2015-12-30 06:56:15.188 | Error on exit 2015-12-30 06:56:15.188 | + [[ -z /opt/stack/logs ]] 2015-12-30 06:56:15.188 | + /home/mspreitz/devstack/tools/worlddump.py -d /opt/stack/logs 2015-12-30 06:56:15.549 | + exit 2 I tested http://docs.openstack.org/developer/devstack/guides/neutron.html#using-neutron-with-a-single-interface with the stable/liberty branch, and stack.sh failed. I used exactly the configuration described in that doc section, with only a link-local IPv6 address on the one network interface, but with a couple of changes to local.conf. One is that I added "IP_VERSION=4" to avoid any attempt to configure Neutron to use IPv6. The other is that I added a few lines to improve the logging --- mainly making the logs go to files (which I will attach to this bug). I used Ubuntu 14.04.3, fully updated. That means kernel 3.19.0-42-generic. I downloaded openvswitch-2.4.0.tar.gz and built the Debian packages for OpenVswitch according to the instruction contained therein. For OVS installation: I eschewed the kernel module I just built, instead relying on what is shipped with Linux 3.19, and I installed the openvswitch-switch and openvswitch-common debian packages I had just built. When I ran stack.sh, it aborted with problems configuring br-ex. In the attached archive, under home/mspreitz/devstack/, you will find a file named typescript, produced by `script`, which shows some of the state before I started stack.sh (as well as a typescript of running stack.sh). You will also find postmortem.log, showing some of the state right after stack.sh aborted. Additionally there is net-recovery.log, showing how I recovered to a working network configuration --- hopefully this gives further clues to what went wrong in stack.sh. You will see that what I did is to change the state of the br-ex link to UP and then add the appropriate route through br-ex (although at least part of that command was wrong and/or unnecessary). You will also find that opt/stack/logs/ has all the logs that stack.sh normally produces. I suspect the problem centers around the br-ex link not being "up". But that's only a slightly informed guess. For easy reference, here is the end of stack.sh.log: 2015-12-30 06:56:15.111 | + IP_BRD='172.18.161.6/24 brd 172.18.161.255' 2015-12-30 06:56:15.111 | + [[ inet == \i\n\e\t\6 ]] 2015-12-30 06:56:15.111 | + '[' 172.18.161.1 '!=' '' ']' 2015-12-30 06:56:15.111 | + ADD_DEFAULT_ROUTE='sudo ip r replace default via 172.18.161.1 dev br-ex' 2015-12-30 06:56:15.111 | + [[ True == \T\r\u\e ]] 2015-12-30 06:56:15.111 | + ADD_OVS_PORT='sudo ovs-vsctl --may-exist add-port br-ex eth0' 2015-12-30 06:56:15.111 | + [[ 172.18.161.6/24 brd 172.18.161.255 != '' ]] 2015-12-30 06:56:15.111 | + IP_DEL='sudo ip addr del 172.18.161.6/24 brd 172.18.161.255 dev eth0' 2015-12-30 06:56:15.111 | + IP_ADD='sudo ip addr add 172.18.161.6/24 brd 172.18.161.255 dev br-ex' 2015-12-30 06:56:15.111 | + sudo ip addr del 172.18.161.6/24 brd 172.18.161.255 dev eth0 2015-12-30 06:56:15.120 | + sudo ip addr add 172.18.161.6/24 brd 172.18.161.255 dev br-ex 2015-12-30 06:56:15.130 | + sudo ovs-vsctl --may-exist add-port br-ex eth0 2015-12-30 06:56:15.173 | + sudo ip r replace default via 172.18.161.1 dev br-ex 2015-12-30 06:56:15.184 | RTNETLINK answers: Network is unreachable 2015-12-30 06:56:15.186 | + exit_trap 2015-12-30 06:56:15.186 | + local r=2 2015-12-30 06:56:15.186 | ++ jobs -p 2015-12-30 06:56:15.187 | + jobs= 2015-12-30 06:56:15.188 | + [[ -n '' ]] 2015-12-30 06:56:15.188 | + kill_spinner 2015-12-30 06:56:15.188 | + '[' '!' -z '' ']' 2015-12-30 06:56:15.188 | + [[ 2 -ne 0 ]] 2015-12-30 06:56:15.188 | + echo 'Error on exit' 2015-12-30 06:56:15.188 | Error on exit 2015-12-30 06:56:15.188 | + [[ -z /opt/stack/logs ]] 2015-12-30 06:56:15.188 | + /home/mspreitz/devstack/tools/worlddump.py -d /opt/stack/logs 2015-12-30 06:56:15.549 | + exit 2