Comment 13 for bug 1823740

TRIAGE:

Confirmed rocky-proposed in use.
Confirmed python3-oslo.cache 1.30.1-0ubuntu1.1~cloud0

The bug is in the relation between nova-cloud-controller and memcached. It is using the wrong space IP address.

Steps to recreate:

When running:
openstack availability zone list
^[Unable to establish connection to http://10.244.40.91:8774/v2.1/os-availability-zone/detail: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

On nova-cloud-conroller/0
root@juju-ab7f6f-18-lxd-5:/var/log/nova# netstat -tn|grep SYN
tcp 0 1 192.168.33.172:39964 192.168.33.163:11211 SYN_SENT

Manual attempt also times out:
root@juju-ab7f6f-18-lxd-5:/var/log/nova# nc -vz 192.168.33.163 11211

ifconfig for nova-cloud-controller/0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
        inet 10.244.41.50 netmask 255.255.248.0 broadcast 10.244.47.255
        inet6 fe80::216:3eff:fe8c:81e2 prefixlen 64 scopeid 0x20<link>
        ether 00:16:3e:8c:81:e2 txqueuelen 1000 (Ethernet)
        RX packets 254198 bytes 295532290 (295.5 MB)
        RX errors 0 dropped 1 overruns 0 frame 0
        TX packets 284144 bytes 109114470 (109.1 MB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
        inet 192.168.33.172 netmask 255.255.255.0 broadcast 192.168.33.255
        inet6 fe80::216:3eff:fe8a:e400 prefixlen 64 scopeid 0x20<link>
        ether 00:16:3e:8a:e4:00 txqueuelen 1000 (Ethernet)
        RX packets 369976 bytes 139660554 (139.6 MB)
        RX errors 0 dropped 1 overruns 0 frame 0
        TX packets 438489 bytes 84210710 (84.2 MB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
        inet 127.0.0.1 netmask 255.0.0.0
        inet6 ::1 prefixlen 128 scopeid 0x10<host>
        loop txqueuelen 1000 (Local Loopback)
        RX packets 151594 bytes 44616315 (44.6 MB)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 151594 bytes 44616315 (44.6 MB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

memcached/1 iptables rules
Chain ufw-user-input (1 references)
target prot opt source destination
ACCEPT tcp -- 10.244.41.55 0.0.0.0/0 tcp dpt:11211
ACCEPT tcp -- 10.244.41.52 0.0.0.0/0 tcp dpt:11211
ACCEPT tcp -- 10.244.41.50 0.0.0.0/0 tcp dpt:11211
ACCEPT tcp -- 192.168.33.180 0.0.0.0/0 tcp dpt:11211
ACCEPT tcp -- 192.168.33.173 0.0.0.0/0 tcp dpt:11211
ACCEPT tcp -- 192.168.33.164 0.0.0.0/0 tcp dpt:11211
ACCEPT tcp -- 192.168.33.167 0.0.0.0/0 tcp dpt:11211
ACCEPT tcp -- 192.168.33.156 0.0.0.0/0 tcp dpt:11211
ACCEPT tcp -- 192.168.33.147 0.0.0.0/0 tcp dpt:11211
ACCEPT tcp -- 192.168.33.151 0.0.0.0/0 tcp dpt:11211
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:22
DROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:11211

juju run --application nova-cloud-controller -- 'ifconfig |egrep "192.168.33|10.244.41"'
- Stdout: |2
            inet 10.244.41.52 netmask 255.255.248.0 broadcast 10.244.47.255
            inet 192.168.33.174 netmask 255.255.255.0 broadcast 192.168.33.255
  UnitId: nova-cloud-controller/1
- Stdout: |2
            inet 10.244.41.55 netmask 255.255.248.0 broadcast 10.244.47.255
            inet 192.168.33.177 netmask 255.255.255.0 broadcast 192.168.33.255
  UnitId: nova-cloud-controller/2
- Stdout: |2
            inet 10.244.41.50 netmask 255.255.248.0 broadcast 10.244.47.255
            inet 192.168.33.172 netmask 255.255.255.0 broadcast 192.168.33.255
  UnitId: nova-cloud-controller/0

Has the wrong space entry for all nova-cloud-controllers: i.e. 10.244.41.50 vs 192.168.33.172

Next steps:
check both nova-cloud-controller and memcached charms to make sure they use the correct space for the relation.