In testrun https://solutions.qa.canonical.com/testruns/777aaded-efec-4d4d-b17b-e600a68521b3, which tests openstack bobcat on jammy, n-c-c stays blocked with the status Services not running that should be: nova-conductor.
The versions used for this run are:
maas 3.3.5-
juju 3.1.6
cpe-foundation 2.21+git.64.g5f71f258
infra-ubuntu focal
charms 2023.2/candidate
fce-container-image ubuntu:jammy
ceph reef/candidate
In the nova-conductor, we see the following message:
===========
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova Traceback (most recent call last):
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova File "/usr/bin/nova-conductor", line 10, in <module>
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova sys.exit(main())
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova File "/usr/lib/python3/dist-packages/nova/cmd/conductor.py", line 45, in main
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova server = service.Service.create(binary='nova-conductor',
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova File "/usr/lib/python3/dist-packages/nova/service.py", line 252, in create
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova service_obj = cls(host, binary, topic, manager,
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova File "/usr/lib/python3/dist-packages/nova/service.py", line 116, in __init__
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova self.manager = manager_class(host=self.host, *args, **kwargs)
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 124, in __init__
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova self.compute_task_mgr = ComputeTaskManager()
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 255, in __init__
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova self.report_client
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 283, in report_client
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova return report.report_client_singleton()
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova File "/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 91, in report_client_singleton
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova PLACEMENTCLIENT = SchedulerReportClient()
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova File "/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 234, in __init__
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova self._client = self._create_client()
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova File "/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 277, in _create_client
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova client = self._adapter or utils.get_sdk_adapter('placement')
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova File "/usr/lib/python3/dist-packages/nova/utils.py", line 995, in get_sdk_adapter
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova return getattr(conn, service_type)
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova File "/usr/lib/python3/dist-packages/openstack/service_description.py", line 89, in __get__
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova proxy = self._make_proxy(instance)
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova File "/usr/lib/python3/dist-packages/openstack/service_description.py", line 265, in _make_proxy
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova found_version = temp_adapter.get_api_major_version()
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 354, in get_api_major_version
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova return self.session.get_api_major_version(auth or self.auth, **kwargs)
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1268, in get_api_major_version
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova return auth.get_api_major_version(self, **kwargs)
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 497, in get_api_major_version
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova data = get_endpoint_data(discover_versions=discover_versions)
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 268, in get_endpoint_data
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova service_catalog = self.get_access(session).service_catalog
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 131, in get_access
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova self.auth_ref = self.get_auth_ref(session)
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/identity/generic/base.py", line 205, in get_auth_ref
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova return self._plugin.get_auth_ref(session, **kwargs)
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/identity/v3/base.py", line 185, in get_auth_ref
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova resp = session.post(token_url, json=body, headers=headers,
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1141, in post
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova return self.request(url, 'POST', **kwargs)
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 978, in request
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova raise exceptions.from_response(resp, method, url)
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova keystoneauth1.exceptions.http.InternalServerError: Internal Server Error (HTTP 500)
nova/nova-conductor.log:2023-11-09 21:01:15.865 322885 ERROR nova
nova/nova-conductor.log:2023-11-09 21:01:17.468 322891 ERROR nova.scheduler.client.report [-] Failed to initialize placement client (is keystone available?): keystoneauth1.exceptions.http.InternalServerError: Internal Server Error (HTTP 500)
nova/nova-conductor.log:2023-11-09 21:01:17.469 322891 ERROR nova.conductor.manager [-] Fatal error initializing placement client: Internal Server Error (HTTP 500): keystoneauth1.exceptions.http.InternalServerError: Internal Server Error (HTTP 500)
===========
I'm not sure where this server error comes from, none of the other services seem to be bothered by it.
Crashdumps and configs can be found here: https://oil-jenkins.canonical.com/artifacts/777aaded-efec-4d4d-b17b-e600a68521b3/index.html
At the same time as the error occurred in the nova-conductor log, the keystone charm was saying:
2023-11-09 21:01:25 DEBUG unit.keystone/
2023-11-09 21:01:25 DEBUG unit.keystone/
2023-11-09 21:01:25 INFO unit.keystone/
keystone/0 shut down shortly later
[Thu Nov 09 21:01:26.068983 2023] [mpm_event:notice] [pid 188079:tid 140702016874368] AH00492: caught SIGWINCH, shutting down gracefully
keystone_2 came on line at 21:02
keystone_0 came on line at 20:46
keystone_1 came on line at 20:56
haproxy logs for the keystone units around 21:01 indicate
keystone_0 was up.
keystone_1 was down.
keystone_2 was up.
---
It seems that nova-conductor just isn't resilient enough when keystone is coming up. It may be that keystone should not be handing out creds, but this is probably a non-tls -> tls bootstrap issue when vault is unsealed.
As this only tends to happen at deployment time, and can be worked around (restart the service), this is not a high priority bug.
In terms of a fix, it's probably necessary that the nova-cc charm attempt to restart nova-conductor if it sees it is 'down' during an update-status hook, or the package just continually tries to restart it.