refused connection to keystone during identity-service-relation-changed

[Marian Gasparovic]

During sunbeam bootstrap keystone failed in identity-service-relation-changed hook.

Keystone log shows many occurrences of
container-agent] keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://localhost:5000/v3/auth/tokens: HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f4be5fa92a0>: Failed to establish a new connection: [Errno 111] Connection refused'))

Link to logs - https://oil-jenkins.canonical.com/artifacts/eb449904-4679-47ed-8a50-d0e86f3d278c/index.html

[James Page (james-page)]

Looking at the log data for keystone/0 the wsgi service failed to start (reason unknown) at which point all subsequent interaction with the service for creation if users fails.

As we've only seen one instance of this bug during testing marking as Low for now.

[James Page (james-page)]

2023-11-25T22:28:32.863Z [container-agent] 2023-11-25 22:28:32 ERROR juju-log amqp:84: Exception raised in section 'Bootstrapping': cannot perform the following tasks:
2023-11-25T22:28:32.863Z [container-agent] - Start service "wsgi-keystone" (cannot start service: exited quickly with code 0)
2023-11-25T22:28:32.863Z [container-agent] ----- Logs from task 1 -----
2023-11-25T22:28:32.863Z [container-agent] 2023-11-25T22:28:32Z INFO Most recent service output:
2023-11-25T22:28:32.863Z [container-agent] (...)
2023-11-25T22:28:32.863Z [container-agent] 2023-11-25 22:28:01.232169 (keystone.common.fernet_utils): 2023-11-25 22:28:01,232 WARNING key_repository is world readable: /etc/keystone/fernet-keys/
2023-11-25T22:28:32.863Z [container-agent] - - - [25/Nov/2023:22:28:01 +0000] "GET /v3/domains HTTP/1.1" 200 753 184897 "-" "python-keystoneclient"
2023-11-25T22:28:32.863Z [container-agent] 2023-11-25 22:28:01.436806 (keystone.common.fernet_utils): 2023-11-25 22:28:01,436 WARNING key_repository is world readable: /etc/keystone/fernet-keys/
2023-11-25T22:28:32.863Z [container-agent] 2023-11-25 22:28:01.653084 (keystone.common.password_hashing): 2023-11-25 22:28:01,652 WARNING Truncating password to algorithm specific maximum length 72 characters.
2023-11-25T22:28:32.863Z [container-agent] - - - [25/Nov/2023:22:28:01 +0000] "POST /v3/users HTTP/1.1" 201 268 458800 "-" "python-keystoneclient"
2023-11-25T22:28:32.863Z [container-agent] 2023-11-25 22:28:01.910132 (keystone.common.fernet_utils): 2023-11-25 22:28:01,910 WARNING key_repository is world readable: /etc/keystone/fernet-keys/
2023-11-25T22:28:32.863Z [container-agent] - - - [25/Nov/2023:22:28:01 +0000] "GET /v3/roles HTTP/1.1" 200 1198 283547 "-" "python-keystoneclient"
2023-11-25T22:28:32.863Z [container-agent] 2023-11-25 22:28:02.233824 (keystone.common.fernet_utils): 2023-11-25 22:28:02,233 WARNING key_repository is world readable: /etc/keystone/fernet-keys/
2023-11-25T22:28:32.863Z [container-agent] - - - [25/Nov/2023:22:28:02 +0000] "GET /v3/domains HTTP/1.1" 200 753 242576 "-" "python-keystoneclient"
2023-11-25T22:28:32.863Z [container-agent] 2023-11-25 22:28:02.510613 (keystone.common.fernet_utils): 2023-11-25 22:28:02,510 WARNING key_repository is world readable: /etc/keystone/fernet-keys/
2023-11-25T22:28:32.863Z [container-agent] - - - [25/Nov/2023:22:28:02 +0000] "GET /v3/users?domain_id=6d201511a1de46b994b66837a2c2e463 HTTP/1.1" 200 1746 206170 "-" "python-keystoneclient"
2023-11-25T22:28:32.863Z [container-agent] 2023-11-25 22:28:02.737966 (keystone.common.fernet_utils): 2023-11-25 22:28:02,737 WARNING key_repository is world readable: /etc/keystone/fernet-keys/
2023-11-25T22:28:32.863Z [container-agent] - - - [25/Nov/2023:22:28:02 +0000] "GET /v3/domains HTTP/1.1" 200 753 238724 "-" "python-keystoneclient"
2023-11-25T22:28:32.863Z [container-agent] 2023-11-25 22:28:02.993937 (keystone.common.fernet_utils): 2023-11-25 22:28:02,993 WARNING key_repository is world readable: /etc/keystone/fernet-keys/
2023-11-25T22:28:32.863Z [container-agent] - - - [25/Nov/2023:22:28:02 +0000] "GET /v3/projects?domain_id=6d201511a1de46b...