nova-network (devstack) network connections to launched instances fail

[John Smith]

When running devstack, I run into the issue where I can successfully launch an instance, but cannot connect to it, not even from the host running devstack. The error is either " Destination Host Unreachable" (when using ping) or "No route to host" (when using ssh).

Steps to reproduce:

./stack.sh
export OS_USERNAME=admin
export OS_PASSWORD=password
export OS_TENANT_NAME=demo
export OS_AUTH_URL=http://192.168.126.142:5000/v2.0/

source /usr/local/src/devstack/openrc admin

nova secgroup-add-rule default tcp 22 22 0.0.0.0/0
nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0

wget https://launchpadlibrarian.net/83303699/cirros-0.3.0-i386-disk.img
glance image-create --name=cirros-0.3.0-i386 --is-public=true --container-format=bare --disk-format=qcow2 < cirros-0.3.0-i386-disk.img
nova boot --flavor m1.nano --image cirros-0.3.0-i386 myvm
nova show myvm
+--------------------------------------+----------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | localhost.localdomain |
| OS-EXT-SRV-ATTR:hypervisor_hostname | localhost.localdomain |
| OS-EXT-SRV-ATTR:instance_name | instance-00000001 |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2013-12-04T18:40:43.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2013-12-04T18:38:26Z |
| flavor | m1.nano (42) |
| hostId | 24de90bc5bb06394e434e1febf1e79f23bcb629f48903b02d028bdf7 |
| id | 452a730e-7fa9-4f2a-9ea1-f792442b8383 |
| image | cirros-0.3.0-i386 (d0a521c9-895f-4046-a85c-5cece89bdfd4) |
| key_name | None |
| metadata | {} |
| name | myvm |
| os-extended-volumes:volumes_attached | [] |
| private network | 10.4.128.2 |
| progress | 0 |
| security_groups | default |
| status | ACTIVE |
| tenant_id | fb8947cbb5f44959af347dc7bdf16e6c |
| updated | 2013-12-04T18:40:43Z |
| user_id | 543a006dea954a05b4249654198681d7 |
+--------------------------------------+----------------------------------------------------------+
$ ping 10.4.128.2
PING 10.4.128.2 (10.4.128.2) 56(84) bytes of data.
From 10.4.128.1 icmp_seq=10 Destination Host Unreachable
From 10.4.128.1 icmp_seq=11 Destination Host Unreachable
From 10.4.128.1 icmp_seq=12 Destination Host Unreachable
From 10.4.128.1 icmp_seq=13 Destination Host Unreachable
^C
--- 10.4.128.2 ping statistics ---
13 packets transmitted, 0 received, +4 errors, 100% packet loss, time 12003ms
pipe 4
$ ssh 10.4.128.2
ssh: connect to host 10.4.128.2 port 22: No route to host
$ ifconfig -a
br100: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
        inet 10.4.128.1 netmask 255.255.240.0 broadcast 10.4.143.255
        inet6 fe80::acc9:71ff:fefb:f36 prefixlen 64 scopeid 0x20<link>
        ether 00:0c:29:6e:32:be txqueuelen 0 (Ethernet)
        RX packets 15649 bytes 6616569 (6.3 MiB)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 22546 bytes 3504448 (3.3 MiB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
        inet 127.0.0.1 netmask 255.0.0.0
        inet6 ::1 prefixlen 128 scopeid 0x10<host>
        loop txqueuelen 0 (Local Loopback)
        RX packets 125621 bytes 144989732 (138.2 MiB)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 125621 bytes 144989732 (138.2 MiB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

p3p1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
        inet6 fe80::20c:29ff:fe6e:32be prefixlen 64 scopeid 0x20<link>
        ether 00:0c:29:6e:32:be txqueuelen 1000 (Ethernet)
        RX packets 68455 bytes 66756968 (63.6 MiB)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 62578 bytes 8182621 (7.8 MiB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
        device interrupt 19 base 0x2000

vnet0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
        inet6 fe80::fc16:3eff:fe6d:4b91 prefixlen 64 scopeid 0x20<link>
        ether fe:16:3e:6d:4b:91 txqueuelen 500 (Ethernet)
        RX packets 23 bytes 2422 (2.3 KiB)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 49 bytes 6309 (6.1 KiB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 192.168.126.2 0.0.0.0 UG 0 0 0 br100
10.4.128.0 * 255.255.240.0 U 0 0 0 br100
192.168.126.0 * 255.255.255.0 U 0 0 0 br100
$ arp -a
? (192.168.126.1) at 00:50:56:c0:00:08 [ether] on br100
? (192.168.126.254) at 00:50:56:eb:f2:f2 [ether] on br100
? (192.168.126.2) at 00:50:56:e6:61:ac [ether] on br100
? (10.4.128.2) at <incomplete> on br100
$ sudo brctl show
bridge name bridge id STP enabled interfaces
br100 8000.000c296e32be no p3p1
                                                        vnet0

[John Smith (lbalbalba)]

Additional info: when I ping the instance (10.4.128.2) from the host I run openstack on (localhost.localdomain, 192.168.126.142) I do see arp request packets on both br100 and vnet0

# tcpdump -i br100 arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br100, link-type EN10MB (Ethernet), capture size 65535 bytes
19:49:23.293070 ARP, Request who-has 10.4.128.2 tell localhost.localdomain, length 28
19:49:24.295030 ARP, Request who-has 10.4.128.2 tell localhost.localdomain, length 28
19:49:25.297188 ARP, Request who-has 10.4.128.2 tell localhost.localdomain, length 28
^C
3 packets captured
3 packets received by filter
0 packets dropped by kernel
# tcpdump -i vnet0 arp
tcpdump: WARNING: vnet0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vnet0, link-type EN10MB (Ethernet), capture size 65535 bytes
19:49:35.404475 ARP, Request who-has 10.4.128.2 tell localhost.localdomain, length 28
19:49:36.406490 ARP, Request who-has 10.4.128.2 tell localhost.localdomain, length 28
19:49:37.409074 ARP, Request who-has 10.4.128.2 tell localhost.localdomain, length 28
^C
3 packets captured
3 packets received by filter
0 packets dropped by kernel

[John Smith (lbalbalba)]

And arp reply packets on br100:
tcpdump -vv -i br100 arp
19:54:48.173534 ARP, Ethernet (len 6), IPv4 (len 4), Reply localhost.localdomain is-at 00:0c:29:6e:32:be (oui Unknown), length 28

[John Smith (lbalbalba)]

Just having a single network interface may have something to do with this issue.