Openstack version: 2023.1
Deployment tool: kayobe
OS: Ubuntu 22.04
After recent update to Antelope (2023.1) we are seeing the following errors in the keystone.log when looking up valid/existing projects:
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application [None ... - - default default] Could not find project: foo.:
keystone.exception.ProjectNotFound: Could not find project: foo.
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application Traceback (most recent call last):
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask/app.py", line 1820, in full_dispatch_request
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application rv = self.dispatch_request()
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask/app.py", line 1796, in dispatch_request
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 467, in wrapper
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application resp = resource(*args, **kwargs)
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask/views.py", line 107, in view
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs)
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 582, in dispatch_request
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application resp = meth(*args, **kwargs)
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/api/projects.py", line 151, in get
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application return self._get_project(project_id)
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/api/projects.py", line 107, in _get_project
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id)
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/manager.py", line 115, in wrapped
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs)
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/decorator.py", line 232, in fun
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw)
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application return self.get_or_create(
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 1042, in get_or_create
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application with Lock(
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/lock.py", line 185, in __enter__
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application return self._enter()
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/lock.py", line 94, in _enter
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime)
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/lock.py", line 178, in _enter_create
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application return self.creator()
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 995, in gen_value
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application created_value = creator(
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/resource/core.py", line 903, in get_project
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application return self.driver.get_project(project_id)
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/resource/backends/sql.py", line 49, in get_project
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict()
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/resource/backends/sql.py", line 44, in _get_project
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id)
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: foo.
We have traced it to a specific openstacksdk call and the issue can be reliably replicated (replace foo by a valid/existing project) by sourcing relevant admin credentials and running:
python -c 'import openstack; conn = openstack.connect(); valid_project = conn.identity.find_project("foo"); print(f"Project id is {valid_project.id}")'
The are no errors returned by the find_project call and correct projectID is displayed, but keystone logs the above error.
