Keystone deploy enters ERROR state with nothing running on port 35337

[Ryan Farrell]

A fresh deployment of Keystone has failed; the charm entered and error state then become blocked with a message indicating it was waiting for a certain number of peers.

2020-08-06 17:15:33 DEBUG identity-service-relation-changed RuntimeError: The call within manager.py failed with the error: 'Unable to establish connection to http://localhost:35337/v3/auth/tokens: HTTPConnectionPool(host='localhost', port=35337): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fda1c1094d0>: Failed to establish a new connection: [Errno 111] Connection refused',))'. The call was: path=['resolve_role_id'], args=('member',), kwargs={}, api_version=None

Logs indicated that there was nothing listing on port 35337, which was confirmed with nc:

# keystone/0 - working unit
ubuntu@juju-2bccdc-9-lxd-1:~$ sudo netstat -tupln | grep 35337
tcp6 0 0 :::35337 :::* LISTEN 30815/apache2

#keystone/1 - non working unit
ubuntu@juju-2bccdc-10-lxd-1:~$ sudo netstat -tupln | grep 35337

Additionally the directory: /etc/apache2/ssl/keystone was completely empty - we expected there to be symlinks to certs / keyst there.

This deployment was using cs:~openstack-charmers-next/keystone-500 - redeploying the unit was successful.