After completing the local multi-node scenario and I left it running in an idle status, eventually mysqlrouter got into a red status.
Making authentication request to http://10.0.123.82/openstack-keystone/v3/auth/tokens
http://10.0.123.82:80 "POST /openstack-keystone/v3/auth/tokens HTTP/1.1" 500 610
Request returned failure status: 500
Internal Server Error (HTTP 500)
Traceback (most recent call last):
File "/snap/openstack/506/lib/python3.10/site-packages/cliff/app.py", line 395, in run_subcommand
self.prepare_to_run_command(cmd)
File "/snap/openstack/506/lib/python3.10/site-packages/osc_lib/shell.py", line 516, in prepare_to_run_command
self.client_manager.auth_ref
File "/snap/openstack/506/lib/python3.10/site-packages/osc_lib/clientmanager.py", line 208, in auth_ref
self._auth_ref = self.auth.get_auth_ref(self.session)
File "/snap/openstack/506/lib/python3.10/site-packages/keystoneauth1/identity/generic/base.py", line 205, in get_auth_ref
return self._plugin.get_auth_ref(session, **kwargs)
File "/snap/openstack/506/lib/python3.10/site-packages/keystoneauth1/identity/v3/base.py", line 185, in get_auth_ref
resp = session.post(token_url, json=body, headers=headers,
File "/snap/openstack/506/lib/python3.10/site-packages/keystoneauth1/session.py", line 1162, in post
return self.request(url, 'POST', **kwargs)
File "/snap/openstack/506/lib/python3.10/site-packages/keystoneauth1/session.py", line 985, in request
raise exceptions.from_response(resp, method, url)
keystoneauth1.exceptions.http.InternalServerError: Internal Server Error (HTTP 500)
clean_up IssueToken: Internal Server Error (HTTP 500)
END return value: 1
$ juju status -m openstack error
Unit Workload Agent Address Ports Message
cinder-ceph-mysql-router/0* error idle 10.1.32.225 hook failed: "update-status"
cinder-ceph-mysql-router/1 error idle 10.1.193.211 hook failed: "update-status"
cinder-ceph-mysql-router/2 error idle 10.1.186.20 hook failed: "update-status"
cinder-mysql-router/0* error idle 10.1.32.218 hook failed: "update-status"
cinder-mysql-router/1 error idle 10.1.193.198 hook failed: "update-status"
cinder-mysql-router/2 error idle 10.1.186.7 hook failed: "update-status"
glance-mysql-router/0* error idle 10.1.32.219 hook failed: "update-status"
glance-mysql-router/1 error idle 10.1.193.194 hook failed: "update-status"
glance-mysql-router/2 error idle 10.1.186.3 hook failed: "update-status"
horizon-mysql-router/0* error idle 10.1.32.209 hook failed: "update-status"
horizon-mysql-router/1 error idle 10.1.186.18 hook failed: "upgrade-version-a-relation-changed"
horizon-mysql-router/2 error idle 10.1.193.209 hook failed: "update-status"
keystone-mysql-router/0* error idle 10.1.32.199 hook failed: "update-status"
...
The Too many connections error was recorded. I'm using 16 CPU cores each on 3 machines fwiw.
root@glance-mysql-router-0:/# cat /var/log/mysqlrouter/archive_mysqlrouter/mysqlrouter.log-20240521_05\:54
2024-05-21 05:53:36 metadata_cache WARNING [7fe0f030b640] Failed connecting with Metadata Server mysql-1.mysql-endpoints.openstack.svc.cluster.local:3306: Too many connections (1040)
2024-05-21 05:53:41 metadata_cache WARNING [7fe0f030b640] While updating metadata, could not establish a connection to cluster 'cluster-6a82396ce20b724018467cb8172c190b' through mysql-1.mysql-endpoints.openstack.svc.cluster.local:3306
