Missing ACL for keystone on internal interface causes openstack service outages

[Michael Skalka]

As seen in this test run: https://solutions.qa.canonical.com/testruns/testRun/a09f8739-60c0-4d0f-ba56-0809e5951f07
Crashdump: https://oil-jenkins.canonical.com/artifacts/a09f8739-60c0-4d0f-ba56-0809e5951f07/generated/generated/openstack/juju-crashdump-openstack-2021-05-11-20.16.50.tar.gz
Bundle: https://oil-jenkins.canonical.com/artifacts/a09f8739-60c0-4d0f-ba56-0809e5951f07/generated/generated/openstack/bundle.yaml
Full artifacts: https://oil-jenkins.canonical.com/artifacts/a09f8739-60c0-4d0f-ba56-0809e5951f07/index.html

This was a Ussuri OpenStack cloud on Focal using the latest stable revision of the charms.

Initial problem with the deployment appears as a nova-scheduler service error putting the nova-cloud-controller unit into a blocked state. The nova-scheduler service failed due to a 500 from Keystone. Looking at the keystone_errors log on the keystone/0 unit we see an ACL issue:

keystone/0 /var/log/apache2/keystone_error.log
...
2021-05-11 17:08:57.001317 File "/usr/lib/python3/dist-packages/pymysql/__init__.py", line 94, in Connect
2021-05-11 17:08:57.001320 return Connection(*args, **kwargs)
2021-05-11 17:08:57.001327 File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 325, in __init__
2021-05-11 17:08:57.001330 self.connect()
2021-05-11 17:08:57.001337 File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 599, in connect
2021-05-11 17:08:57.001340 self._request_authentication()
2021-05-11 17:08:57.001347 File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 871, in _request_authentication
2021-05-11 17:08:57.001350 auth_packet = self._process_auth(plugin_name, auth_packet)
2021-05-11 17:08:57.001361 File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 902, in _process_auth
2021-05-11 17:08:57.001364 return _auth.sha256_password_auth(self, auth_packet)
2021-05-11 17:08:57.001371 File "/usr/lib/python3/dist-packages/pymysql/_auth.py", line 183, in sha256_password_auth
2021-05-11 17:08:57.001374 return _roundtrip(conn, data)
2021-05-11 17:08:57.001381 File "/usr/lib/python3/dist-packages/pymysql/_auth.py", line 122, in _roundtrip
2021-05-11 17:08:57.001384 pkt = conn._read_packet()
2021-05-11 17:08:57.001391 File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 684, in _read_packet
2021-05-11 17:08:57.001394 packet.check_error()
2021-05-11 17:08:57.001401 File "/usr/lib/python3/dist-packages/pymysql/protocol.py", line 220, in check_error
2021-05-11 17:08:57.001404 err.raise_mysql_exception(self._data)
2021-05-11 17:08:57.001411 File "/usr/lib/python3/dist-packages/pymysql/err.py", line 109, in raise_mysql_exception
2021-05-11 17:08:57.001414 raise errorclass(errno, errval)
2021-05-11 17:08:57.001434 sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1045, "Access denied for user 'keystone'@'192.168.33.81' (using password: YES)")
2021-05-11 17:08:57.001438 (Background on this error at: http://sqlalche.me/e/e3q8)

Meaning that keystone unit did not get the correct access set up on its internal binding for mysql. The other two units in the cluster don't appear to have this issue (empty keystone_error log).

[Konstantinos Kaskavelis (kaskavel)]

Closing this due to inactivity (low number of occurrences, and no hit for more than one year)