keystone cannot connect to itelf during resize

[Marian Gasparovic]

keystone charm goes into error during sunbeam cluster resize reporting `hook failed: "ingress-public-relation-changed"`

pod logs show

 keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://localhost:5000/v3/auth/tokens: HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff811a62c80>: Failed to establish a new connection: [Errno 111] Connection refused'))

[Marian Gasparovic (marosg)]

logs - https://oil-jenkins.canonical.com/artifacts/0a7cbd1e-872d-47bc-9c0f-984ee392ccca/index.html

[Nobuto Murata (nobuto)]

In my case,

$ sunbeam cluster resize
⠴ Deploying OpenStack Control Plane to Kubernetes (this may take a
while) ... waiting for services to come online (72/73)Timed out while wa
Error: Timed out while waiting for model 'openstack' to be ready

real 90m20.187s
user 5m42.335s
sys 0m21.470s

keystone/1 error idle 10.1.193.224 hook failed: "ingress-public-relation-changed"

unit-keystone-1: 2024-05-16 22:36:56 DEBUG unit.keystone/1.juju-log ingress-public:66: Emitting Juju event ingress_public_relation_changed.
unit-keystone-1: 2024-05-16 22:36:56 WARNING unit.keystone/1.juju-log ingress-public:66: 'app' expected but not received.
unit-keystone-1: 2024-05-16 22:36:56 WARNING unit.keystone/1.juju-log ingress-public:66: 'app_name' expected in snapshot but not found.
unit-keystone-1: 2024-05-16 22:36:56 DEBUG unit.keystone/1.juju-log ingress-public:66: Emitting custom event <IngressPerAppReadyEvent via KeystoneOperatorCharm/IngressPerAppRequirer
[ingress-public]/on/ready[130]>.
unit-keystone-1: 2024-05-16 22:36:56 DEBUG unit.keystone/1.juju-log ingress-public:66: Received url: http://10.0.123.81/openstack-keystone
unit-keystone-1: 2024-05-16 22:36:56 DEBUG unit.keystone/1.juju-log ingress-public:66: Received an ingress_changed event
unit-keystone-1: 2024-05-16 22:36:56 DEBUG unit.keystone/1.juju-log ingress-public:66: Keys found
unit-keystone-1: 2024-05-16 22:36:56 DEBUG unit.keystone/1.juju-log ingress-public:66: Making authentication request to http://localhost:5000/v3/auth/tokens
unit-keystone-1: 2024-05-16 22:36:56 DEBUG unit.keystone/1.juju-log ingress-public:66: Starting new HTTP connection (1): localhost:5000
unit-keystone-1: 2024-05-16 22:36:56 ERROR unit.keystone/1.juju-log ingress-public:66: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-keystone-1/charm/venv/urllib3/connection.py", line 198, in _new_conn
    sock = connection.create_connection(
  File "/var/lib/juju/agents/unit-keystone-1/charm/venv/urllib3/util/connection.py", line 85, in create_connection
    raise err
  File "/var/lib/juju/agents/unit-keystone-1/charm/venv/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

[Nobuto Murata (nobuto)]

There was no apache2 process in keystone/1 in my case.

$ juju ssh -m openstack --container keystone keystone/1 ps faux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 149 0.0 0.0 2908 996 pts/1 Ss+ 01:22 0:00 sh -c ps faux
root 156 0.0 0.0 7080 1644 pts/1 R+ 01:22 0:00 \_ ps faux
root 1 0.0 0.0 1234316 10588 ? Ssl May16 0:07 /charm/bin/pebble run --create-dirs --hold --http :38813 --verbose

$ juju ssh -m openstack --container keystone keystone/0 ps faux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 18220 0.0 0.0 2900 984 pts/0 Ss+ 01:22 0:00 sh -c ps faux
root 18227 0.0 0.0 7072 1620 pts/0 R+ 01:22 0:00 \_ ps faux
root 1 0.0 0.0 1235588 11496 ? Ssl May16 0:21 /charm/bin/pebble run --create-dirs --hold --http :38813 --verbose
root 133 0.0 0.0 2900 968 ? S May16 0:00 /bin/sh /usr/sbin/apache2ctl -DFOREGROUND
root 135 0.0 0.0 97280 14408 ? S May16 0:01 \_ /usr/sbin/apache2 -DFOREGROUND
keystone 136 0.3 0.7 335012 114956 ? Sl May16 1:34 \_ (wsgi:keystone-pu -DFOREGROUND
keystone 137 0.4 0.7 335012 114836 ? Sl May16 1:45 \_ (wsgi:keystone-pu -DFOREGROUND
keystone 138 0.4 0.7 337060 117352 ? Sl May16 1:48 \_ (wsgi:keystone-pu -DFOREGROUND
keystone 139 0.4 0.7 335012 114872 ? Sl May16 1:49 \_ (wsgi:keystone-pu -DFOREGROUND
www-data 140 0.0 0.0 2093444 14180 ? Sl May16 0:02 \_ /usr/sbin/apache2 -DFOREGROUND
www-data 141 0.0 0.0 2093524 13464 ? Sl May16 0:01 \_ /usr/sbin/apache2 -DFOREGROUND