Keystone unreachable from the overcloud

[Facundo Ciccioli]

Hi there!

We've observed a situation in which Keystone was unreachable from the overcloud, while still functioning from the undercloud.

The container had this interfaces and routes configured:

ubuntu@juju-40ffba-2-lxd-27:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: gre0@NONE: <NOARP> mtu 1476 qdisc noop state DOWN group default qlen 1
    link/gre 0.0.0.0 brd 0.0.0.0
3: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
189: eth0@if190: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:30:27:1b brd ff:ff:ff:ff:ff:ff
    inet 10.101.140.45/24 brd 10.101.140.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fe30:271b/64 scope link
       valid_lft forever preferred_lft forever
191: eth1@if192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:63:ff:87 brd ff:ff:ff:ff:ff:ff
    inet 10.101.146.6/24 brd 10.101.146.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fe63:ff87/64 scope link
       valid_lft forever preferred_lft forever

ubuntu@juju-40ffba-2-lxd-27:~$ ip r
default via 10.101.140.254 dev eth0
10.101.140.0/24 dev eth0 proto kernel scope link src 10.101.140.45
10.101.146.0/24 dev eth1 proto kernel scope link src 10.101.146.6

There was another container running keystone that was reachable, this one having a single interface:

ubuntu@juju-machine-0-lxc-7:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: gre0@NONE: <NOARP> mtu 1476 qdisc noop state DOWN group default qlen 1
    link/gre 0.0.0.0 brd 0.0.0.0
3: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
92: eth0@if93: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8000 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:82:e8:93 brd ff:ff:ff:ff:ff:ff
    inet 10.101.140.172/24 brd 10.101.140.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.101.140.240/24 brd 10.101.140.255 scope global secondary eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fe82:e893/64 scope link
       valid_lft forever preferred_lft forever

ubuntu@juju-machine-0-lxc-7:~$ ip r
default via 10.101.140.254 dev eth0
10.101.140.0/24 dev eth0 proto kernel scope link src 10.101.140.172
10.101.146.0/24 via 10.101.140.249 dev eth0

It would be nice to have some monitoring for detecting this type of situation in the future.

Thanks!

[Chris Sanders (chris.sanders)]

After reviewing this bug, the issue appears to be a routing/networking issue and I don't see where keystone could reasonable verify that in the charm. I'll close this bug.