Ubuntu
magnum package

  1. Focal (20.04)
  2. Bug #2039599
  3. Activity log

Activity log for bug #2039599

Date Who What changed Old value New value Message
2023-10-17 19:47:30 Felipe Reyes bug added bug
2023-10-17 19:47:45 Felipe Reyes nominated for series Ubuntu Focal
2023-10-17 19:47:45 Felipe Reyes bug task added magnum (Ubuntu Focal)
2023-10-17 19:47:51 Felipe Reyes magnum (Ubuntu): status New Invalid
2023-10-17 19:55:41 Felipe Reyes description [Impact] When deleting a cluster (from the web dashboard or CLI), the cluster stays in DELETE_IN_PROGRESS , internally magnum-conductor issues calls to heat to delete the stacks, although the periodic job that tracks the progress fails when the trust has been deleted, this prevents magnum-conductor to query Heat. Stacktrace: 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall [req-5d099ed5-c878-454f-91b7-6979bd674f62 - - - - -] Fixed interval looping call 'magnum.service.periodic.ClusterUpdateJob.update_status' failed: magnum.common.exception.AuthorizationFailure: unexpected keystone client error occurred: You are not authorized to perform the requested action. (HTTP 403) (Request-ID: req-6bfd2d88-87fa-4d21-959e-e8e994077823) 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall Traceback (most recent call last): 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/magnum/common/exception.py", line 57, in wrapped 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall return func(*args, **kw) 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/magnum/common/clients.py", line 110, in heat 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall endpoint = self.url_for(service_type='orchestration', 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/magnum/common/clients.py", line 48, in url_for 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall return self.keystone().session.get_endpoint(**kwargs) 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1225, in get_endpoint 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall return auth.get_endpoint(self, **kwargs) 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 375, in get_endpoint 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall endpoint_data = self.get_endpoint_data( 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall service_catalog = self.get_access(session).service_catalog 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall self.auth_ref = self.get_auth_ref(session) 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/keystoneauth1/identity/v3/base.py", line 183, in get_auth_ref 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall resp = session.post(token_url, json=body, headers=headers, 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1131, in post 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall return self.request(url, 'POST', **kwargs) 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 968, in request 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall raise exceptions.from_response(resp, method, url) 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall keystoneauth1.exceptions.http.Forbidden: You are not authorized to perform the requested action. (HTTP 403) (Request-ID: req-6bfd2d88-87fa-4d21-959e-e8e994077823) 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall During handling of the above exception, another exception occurred: 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall Traceback (most recent call last): 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/oslo_service/loopingcall.py", line 150, in _run_loop 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall result = func(*self.args, **self.kw) 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/magnum/service/periodic.py", line 71, in update_status 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall cdriver.update_cluster_status(self.ctx, self.cluster) 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/magnum/drivers/heat/driver.py", line 131, in update_cluster_status 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall poller.poll_and_check() 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/magnum/drivers/heat/driver.py", line 470, in poll_and_check 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall status = self.extract_nodegroup_status() 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/magnum/drivers/heat/driver.py", line 495, in extract_nodegroup_status 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall stack = self.openstack_client.heat().stacks.get( 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/magnum/common/exception.py", line 62, in wrapped 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall raise AuthorizationFailure( 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall magnum.common.exception.AuthorizationFailure: unexpected keystone client error occurred: You are not authorized to perform the requested action. (HTTP 403) (Request-ID: req-6bfd2d88-87fa-4d21-959e-e8e994077823) [Test Plan] TBD [Where problems could occur] TBD [Other Info] - Upstream bug: https://storyboard.openstack.org/#!/story/2010232 [Impact] When deleting a cluster (from the web dashboard or CLI), the cluster stays in DELETE_IN_PROGRESS , internally magnum-conductor issues calls to heat to delete the stacks, although the periodic job that tracks the progress fails when the trust has been deleted, this prevents magnum-conductor to query Heat. Stacktrace: 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall [req-5d099ed5-c878-454f-91b7-6979bd674f62 - - - - -] Fixed interval looping call 'magnum.service.periodic.ClusterUpdateJob.update_status' failed: magnum.common.exception.AuthorizationFailure: unexpected keystone client error occurred: You are not authorized to perform the requested action. (HTTP 403) (Request-ID: req-6bfd2d88-87fa-4d21-959e-e8e994077823) 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall Traceback (most recent call last): 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/magnum/common/exception.py", line 57, in wrapped 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall return func(*args, **kw) 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/magnum/common/clients.py", line 110, in heat 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall endpoint = self.url_for(service_type='orchestration', 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/magnum/common/clients.py", line 48, in url_for 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall return self.keystone().session.get_endpoint(**kwargs) 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1225, in get_endpoint 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall return auth.get_endpoint(self, **kwargs) 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 375, in get_endpoint 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall endpoint_data = self.get_endpoint_data( 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall service_catalog = self.get_access(session).service_catalog 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall self.auth_ref = self.get_auth_ref(session) 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/keystoneauth1/identity/v3/base.py", line 183, in get_auth_ref 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall resp = session.post(token_url, json=body, headers=headers, 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1131, in post 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall return self.request(url, 'POST', **kwargs) 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 968, in request 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall raise exceptions.from_response(resp, method, url) 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall keystoneauth1.exceptions.http.Forbidden: You are not authorized to perform the requested action. (HTTP 403) (Request-ID: req-6bfd2d88-87fa-4d21-959e-e8e994077823) 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall During handling of the above exception, another exception occurred: 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall Traceback (most recent call last): 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/oslo_service/loopingcall.py", line 150, in _run_loop 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall result = func(*self.args, **self.kw) 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/magnum/service/periodic.py", line 71, in update_status 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall cdriver.update_cluster_status(self.ctx, self.cluster) 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/magnum/drivers/heat/driver.py", line 131, in update_cluster_status 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall poller.poll_and_check() 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/magnum/drivers/heat/driver.py", line 470, in poll_and_check 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall status = self.extract_nodegroup_status() 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/magnum/drivers/heat/driver.py", line 495, in extract_nodegroup_status 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall stack = self.openstack_client.heat().stacks.get( 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/magnum/common/exception.py", line 62, in wrapped 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall raise AuthorizationFailure( 2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall magnum.common.exception.AuthorizationFailure: unexpected keystone client error occurred: You are not authorized to perform the requested action. (HTTP 403) (Request-ID: req-6bfd2d88-87fa-4d21-959e-e8e994077823) [Test Plan] TBD [Where problems could occur] TBD [Other Info] Upstream bug: https://storyboard.openstack.org/#!/story/2010232 Fixed by: https://opendev.org/openstack/magnum/commit/1ed78a44384ae2617e49acb9031fa3f0964f12f7 This bug was fixed in upstream during the OpenStack Zed (Ubuntu 22.10) development cycle and backported to OpenStack Yoga, Xena and Wallaby Ubuntu 22.04 has the magnum-14.1.0 which corresponds to OpenStack Yoga, the fix backported ( https://opendev.org/openstack/magnum/commit/4d30bef60227cd8246f4caa00857761fa56b7d51 ) is available $ git tag --contains 4d30bef60227cd8246f4caa00857761fa56b7d51 14.1.0 14.1.1 $ rmadison magnum | grep jammy magnum | 14.0.0-0ubuntu1 | jammy/universe | source magnum | 14.1.0-0ubuntu1 | jammy-updates/universe | source magnum | 14.1.1-0ubuntu1 | jammy-proposed/universe | source
2023-10-17 19:55:49 Felipe Reyes magnum (Ubuntu Focal): assignee Felipe Reyes (freyes)
2023-10-17 19:56:15 Felipe Reyes summary Fixed interval looping call 'magnum.service.periodic.ClusterUpdateJob.update_status' failed: magnum.common.exception.AuthorizationFailure: unexpected keystone client error occurred: You are not authorized to perform the requested action Fixed interval looping call 'magnum.service.periodic.ClusterUpdateJob.update_status' failed: magnum.common.exception.AuthorizationFailure
2023-10-17 20:42:02 Felipe Reyes magnum (Ubuntu Focal): importance Undecided High
2023-10-17 20:42:08 Felipe Reyes magnum (Ubuntu Focal): status New In Progress
2023-10-17 20:42:19 Felipe Reyes summary Fixed interval looping call 'magnum.service.periodic.ClusterUpdateJob.update_status' failed: magnum.common.exception.AuthorizationFailure [SRU] Fixed interval looping call 'magnum.service.periodic.ClusterUpdateJob.update_status' failed: magnum.common.exception.AuthorizationFailure
2023-10-18 13:13:51 Felipe Reyes summary [SRU] Fixed interval looping call 'magnum.service.periodic.ClusterUpdateJob.update_status' failed: magnum.common.exception.AuthorizationFailure Fixed interval looping call 'magnum.service.periodic.ClusterUpdateJob.update_status' failed: magnum.common.exception.AuthorizationFailure