Fixed interval looping call 'magnum.service.periodic.ClusterUpdateJob.update_status' failed: magnum.common.exception.AuthorizationFailure

[Felipe Reyes]

[Impact]

When deleting a cluster (from the web dashboard or CLI), the cluster stays in DELETE_IN_PROGRESS , internally magnum-conductor issues calls to heat to delete the stacks, although the periodic job that tracks the progress fails when the trust has been deleted, this prevents magnum-conductor to query Heat.

Stacktrace:

2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall [req-5d099ed5-c878-454f-91b7-6979bd674f62 - - - - -] Fixed interval looping call 'magnum.service.periodic.ClusterUpdateJob.update_status' failed: magnum.common.exception.AuthorizationFailure: unexpected keystone client error occurred: You are not authorized to perform the requested action. (HTTP 403) (Request-ID: req-6bfd2d88-87fa-4d21-959e-e8e994077823)
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall Traceback (most recent call last):
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/magnum/common/exception.py", line 57, in wrapped
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall return func(*args, **kw)
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/magnum/common/clients.py", line 110, in heat
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall endpoint = self.url_for(service_type='orchestration',
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/magnum/common/clients.py", line 48, in url_for
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall return self.keystone().session.get_endpoint(**kwargs)
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1225, in get_endpoint
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall return auth.get_endpoint(self, **kwargs)
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 375, in get_endpoint
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall endpoint_data = self.get_endpoint_data(
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall service_catalog = self.get_access(session).service_catalog
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall self.auth_ref = self.get_auth_ref(session)
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/keystoneauth1/identity/v3/base.py", line 183, in get_auth_ref
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall resp = session.post(token_url, json=body, headers=headers,
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1131, in post
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall return self.request(url, 'POST', **kwargs)
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 968, in request
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall raise exceptions.from_response(resp, method, url)
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall keystoneauth1.exceptions.http.Forbidden: You are not authorized to perform the requested action. (HTTP 403) (Request-ID: req-6bfd2d88-87fa-4d21-959e-e8e994077823)
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall During handling of the above exception, another exception occurred:
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall Traceback (most recent call last):
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/oslo_service/loopingcall.py", line 150, in _run_loop
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall result = func(*self.args, **self.kw)
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/magnum/service/periodic.py", line 71, in update_status
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall cdriver.update_cluster_status(self.ctx, self.cluster)
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/magnum/drivers/heat/driver.py", line 131, in update_cluster_status
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall poller.poll_and_check()
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/magnum/drivers/heat/driver.py", line 470, in poll_and_check
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall status = self.extract_nodegroup_status()
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/magnum/drivers/heat/driver.py", line 495, in extract_nodegroup_status
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall stack = self.openstack_client.heat().stacks.get(
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/magnum/common/exception.py", line 62, in wrapped
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall raise AuthorizationFailure(
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall magnum.common.exception.AuthorizationFailure: unexpected keystone client error occurred: You are not authorized to perform the requested action. (HTTP 403) (Request-ID: req-6bfd2d88-87fa-4d21-959e-e8e994077823)

[Test Plan]

TBD

[Where problems could occur]

TBD

[Other Info]

Upstream bug: https://storyboard.openstack.org/#!/story/2010232
Fixed by: https://opendev.org/openstack/magnum/commit/1ed78a44384ae2617e49acb9031fa3f0964f12f7

This bug was fixed in upstream during the OpenStack Zed (Ubuntu 22.10) development cycle and backported to OpenStack Yoga, Xena and Wallaby

Ubuntu 22.04 has the magnum-14.1.0 which corresponds to OpenStack Yoga, the fix backported ( https://opendev.org/openstack/magnum/commit/4d30bef60227cd8246f4caa00857761fa56b7d51 ) is available

$ git tag --contains 4d30bef60227cd8246f4caa00857761fa56b7d51
14.1.0
14.1.1
$ rmadison magnum | grep jammy
 magnum | 14.0.0-0ubuntu1 | jammy/universe | source
 magnum | 14.1.0-0ubuntu1 | jammy-updates/universe | source
 magnum | 14.1.1-0ubuntu1 | jammy-proposed/universe | source

[Felipe Reyes (freyes)]

PPA for testing purposes available at https://launchpad.net/~freyes/+archive/ubuntu/lp2039599

[Felipe Reyes (freyes)]

I've been reading the patch and I find something different between the expectations of failure in
the patch and the stacktrace posted in the description.

Specifically the patch contains this line:

+ except exception.AuthorizationFailure as e:
+ trust_ex = ("Could not find trust: %s" % self.cluster.trust_id)
[...]
+ if trust_ex in str(e):
[...]

While the error in the stacktrace is not about "could not find trust", it's simply a "you are not
authorized to perform the requested action".

magnum.common.exception.AuthorizationFailure: unexpected keystone client error occurred: You are not
authorized to perform the requested action. (HTTP 403)

I will try to get copies of the keystone logs for 2023-09-08 to know why keystone didn't allow the
access.