2023-07-13 06:13:28.436 1095570 WARNING nova.conductor.api [req-ccd0cb45-a009-45ba-ad2e-7bd8f9dd1e2a - - - - -] Timed out waiting for nova-conductor. Is it running? Or did this service start before nova-conductor? Reattempting establishment of nova-conductor connection...: oslo_m
essaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 578237a0fd7b466babfdf5fb382f49d9
2023-07-13 06:14:08.622 1133261 CRITICAL nova [req-31315d1a-f5d4-4fa9-80d1-96fbe385941f - - - - -] Unhandled error: keystoneauth1.exceptions.catalog.EndpointNotFound: ['internal', 'public'] endpoint for placement service in RegionOne region not found
2023-07-13 06:14:08.622 1133261 ERROR nova Traceback (most recent call last):
2023-07-13 06:14:08.622 1133261 ERROR nova File "/usr/bin/nova-compute", line 10, in <module>
2023-07-13 06:14:08.622 1133261 ERROR nova sys.exit(main())
2023-07-13 06:14:08.622 1133261 ERROR nova File "/usr/lib/python3/dist-packages/nova/cmd/compute.py", line 59, in main
2023-07-13 06:14:08.622 1133261 ERROR nova server = service.Service.create(binary='nova-compute',
2023-07-13 06:14:08.622 1133261 ERROR nova File "/usr/lib/python3/dist-packages/nova/service.py", line 252, in create
2023-07-13 06:14:08.622 1133261 ERROR nova service_obj = cls(host, binary, topic, manager,
2023-07-13 06:14:08.622 1133261 ERROR nova File "/usr/lib/python3/dist-packages/nova/service.py", line 116, in __init__
2023-07-13 06:14:08.622 1133261 ERROR nova self.manager = manager_class(host=self.host, *args, **kwargs)
2023-07-13 06:14:08.622 1133261 ERROR nova File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 631, in __init__
2023-07-13 06:14:08.622 1133261 ERROR nova self.reportclient = report.SchedulerReportClient()
2023-07-13 06:14:08.622 1133261 ERROR nova File "/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 234, in __init__
2023-07-13 06:14:08.622 1133261 ERROR nova self._client = self._create_client()
2023-07-13 06:14:08.622 1133261 ERROR nova File "/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 277, in _create_client
2023-07-13 06:14:08.622 1133261 ERROR nova client = self._adapter or utils.get_sdk_adapter('placement')
2023-07-13 06:14:08.622 1133261 ERROR nova File "/usr/lib/python3/dist-packages/nova/utils.py", line 985, in get_sdk_adapter
2023-07-13 06:14:08.622 1133261 ERROR nova return getattr(conn, service_type)
2023-07-13 06:14:08.622 1133261 ERROR nova File "/usr/lib/python3/dist-packages/openstack/service_description.py", line 87, in __get__
2023-07-13 06:14:08.622 1133261 ERROR nova proxy = self._make_proxy(instance)
2023-07-13 06:14:08.622 1133261 ERROR nova File "/usr/lib/python3/dist-packages/openstack/service_description.py", line 262, in _make_proxy
2023-07-13 06:14:08.622 1133261 ERROR nova found_version = temp_adapter.get_api_major_version()
2023-07-13 06:14:08.622 1133261 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 354, in get_api_major_version
2023-07-13 06:14:08.622 1133261 ERROR nova return self.session.get_api_major_version(auth or self.auth, **kwargs)
2023-07-13 06:14:08.622 1133261 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1276, in get_api_major_version
2023-07-13 06:14:08.622 1133261 ERROR nova return auth.get_api_major_version(self, **kwargs)
2023-07-13 06:14:08.622 1133261 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 500, in get_api_major_version
2023-07-13 06:14:08.622 1133261 ERROR nova data = get_endpoint_data(discover_versions=discover_versions)
2023-07-13 06:14:08.622 1133261 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 275, in get_endpoint_data
2023-07-13 06:14:08.622 1133261 ERROR nova endpoint_data = service_catalog.endpoint_data_for(
2023-07-13 06:14:08.622 1133261 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/access/service_catalog.py", line 462, in endpoint_data_for
2023-07-13 06:14:08.622 1133261 ERROR nova raise exceptions.EndpointNotFound(msg)
2023-07-13 06:14:08.622 1133261 ERROR nova keystoneauth1.exceptions.catalog.EndpointNotFound: ['internal', 'public'] endpoint for placement service in RegionOne region not found
2023-07-13 06:36:36.692 1474436 WARNING keystoneauth.identity.generic.base [req-7609443e-0609-4428-97f8-6f5b41fc00d4 - - - - -] Failed to discover available identity versions when contacting http://keystone-internal.silo4.lab1.solutionsqa:35357. Attempting to parse version from URL
.: keystoneauth1.exceptions.http.BadRequest: Bad Request (HTTP 400)
2023-07-13 06:36:36.696 1474436 CRITICAL nova [req-7609443e-0609-4428-97f8-6f5b41fc00d4 - - - - -] Unhandled error: keystoneauth1.exceptions.discovery.DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is c
orrect. Bad Request (HTTP 400)
2023-07-13 06:36:36.696 1474436 ERROR nova Traceback (most recent call last):
2023-07-13 06:36:36.696 1474436 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/identity/generic/base.py", line 136, in _do_create_plugin
2023-07-13 06:36:36.696 1474436 ERROR nova disc = self.get_discovery(session,
2023-07-13 06:36:36.696 1474436 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 608, in get_discovery
2023-07-13 06:36:36.696 1474436 ERROR nova return discover.get_discovery(session=session, url=url,
2023-07-13 06:36:36.696 1474436 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/discover.py", line 1452, in get_discovery
2023-07-13 06:36:36.696 1474436 ERROR nova disc = Discover(session, url, authenticated=authenticated)
2023-07-13 06:36:36.696 1474436 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/discover.py", line 535, in __init__
2023-07-13 06:36:36.696 1474436 ERROR nova self._data = get_version_data(session, url,
2023-07-13 06:36:36.696 1474436 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/discover.py", line 102, in get_version_data
2023-07-13 06:36:36.696 1474436 ERROR nova resp = session.get(url, headers=headers, authenticated=authenticated)
2023-07-13 06:36:36.696 1474436 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1141, in get
2023-07-13 06:36:36.696 1474436 ERROR nova return self.request(url, 'GET', **kwargs)
2023-07-13 06:36:36.696 1474436 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 986, in request
2023-07-13 06:36:36.696 1474436 ERROR nova raise exceptions.from_response(resp, method, url)
2023-07-13 06:36:36.696 1474436 ERROR nova keystoneauth1.exceptions.http.BadRequest: Bad Request (HTTP 400)
SQA saw this in some of recent Yoga runs, posting some links and error below. /solutions. qa.canonical. com/testruns/ 94e7f7c0- 6ac7-490b- a349-6048851aed 10 /solutions. qa.canonical. com/testruns/ 7c5c3299- 895a-4da3- 93ab-fc0945eee4 c0 /solutions. qa.canonical. com/testruns/ c09547ef- 0fb0-4a7e- bb85-62fe4c8e08 5d
https:/
https:/
https:/
We have some config issue about cpu-shared-set, cpu-dedicated-set. Not sure if they are related.
Will fix this see.
Errors from nova-compute.log
2023-07-13 05:11:19.100 34790 ERROR oslo.messaging. _drivers. impl_rabbit [req-a12e35e8- e3a8-4b1d- bfa0-1709fd2f93 8c - - - - -] Connection failed: [Errno 111] ECONNREFUSED (retrying in 9.0 seconds): ConnectionRefus edError: [Errno 111] ECONNREFUSED
2023-07-13 06:13:28.436 1095570 WARNING nova.conductor.api [req-ccd0cb45- a009-45ba- ad2e-7bd8f9dd1e 2a - - - - -] Timed out waiting for nova-conductor. Is it running? Or did this service start before nova-conductor? Reattempting establishment of nova-conductor connection...: oslo_m exceptions. MessagingTimeou t: Timed out waiting for a reply to message ID 578237a0fd7b466 babfdf5fb382f49 d9
essaging.
2023-07-13 06:14:08.622 1133261 CRITICAL nova [req-31315d1a- f5d4-4fa9- 80d1-96fbe38594 1f - - - - -] Unhandled error: keystoneauth1. exceptions. catalog. EndpointNotFoun d: ['internal', 'public'] endpoint for placement service in RegionOne region not found nova-compute" , line 10, in <module> python3/ dist-packages/ nova/cmd/ compute. py", line 59, in main Service. create( binary= 'nova-compute' , python3/ dist-packages/ nova/service. py", line 252, in create python3/ dist-packages/ nova/service. py", line 116, in __init__ class(host= self.host, *args, **kwargs) python3/ dist-packages/ nova/compute/ manager. py", line 631, in __init__ SchedulerReport Client( ) python3/ dist-packages/ nova/scheduler/ client/ report. py", line 234, in __init__ client( ) python3/ dist-packages/ nova/scheduler/ client/ report. py", line 277, in _create_client sdk_adapter( 'placement' ) python3/ dist-packages/ nova/utils. py", line 985, in get_sdk_adapter python3/ dist-packages/ openstack/ service_ description. py", line 87, in __get__ proxy(instance) python3/ dist-packages/ openstack/ service_ description. py", line 262, in _make_proxy get_api_ major_version( ) python3/ dist-packages/ keystoneauth1/ adapter. py", line 354, in get_api_ major_version get_api_ major_version( auth or self.auth, **kwargs) python3/ dist-packages/ keystoneauth1/ session. py", line 1276, in get_api_ major_version api_major_ version( self, **kwargs) python3/ dist-packages/ keystoneauth1/ identity/ base.py" , line 500, in get_api_ major_version data(discover_ versions= discover_ versions) python3/ dist-packages/ keystoneauth1/ identity/ base.py" , line 275, in get_endpoint_data catalog. endpoint_ data_for( python3/ dist-packages/ keystoneauth1/ access/ service_ catalog. py", line 462, in endpoint_data_for EndpointNotFoun d(msg) exceptions. catalog. EndpointNotFoun d: ['internal', 'public'] endpoint for placement service in RegionOne region not found
2023-07-13 06:14:08.622 1133261 ERROR nova Traceback (most recent call last):
2023-07-13 06:14:08.622 1133261 ERROR nova File "/usr/bin/
2023-07-13 06:14:08.622 1133261 ERROR nova sys.exit(main())
2023-07-13 06:14:08.622 1133261 ERROR nova File "/usr/lib/
2023-07-13 06:14:08.622 1133261 ERROR nova server = service.
2023-07-13 06:14:08.622 1133261 ERROR nova File "/usr/lib/
2023-07-13 06:14:08.622 1133261 ERROR nova service_obj = cls(host, binary, topic, manager,
2023-07-13 06:14:08.622 1133261 ERROR nova File "/usr/lib/
2023-07-13 06:14:08.622 1133261 ERROR nova self.manager = manager_
2023-07-13 06:14:08.622 1133261 ERROR nova File "/usr/lib/
2023-07-13 06:14:08.622 1133261 ERROR nova self.reportclient = report.
2023-07-13 06:14:08.622 1133261 ERROR nova File "/usr/lib/
2023-07-13 06:14:08.622 1133261 ERROR nova self._client = self._create_
2023-07-13 06:14:08.622 1133261 ERROR nova File "/usr/lib/
2023-07-13 06:14:08.622 1133261 ERROR nova client = self._adapter or utils.get_
2023-07-13 06:14:08.622 1133261 ERROR nova File "/usr/lib/
2023-07-13 06:14:08.622 1133261 ERROR nova return getattr(conn, service_type)
2023-07-13 06:14:08.622 1133261 ERROR nova File "/usr/lib/
2023-07-13 06:14:08.622 1133261 ERROR nova proxy = self._make_
2023-07-13 06:14:08.622 1133261 ERROR nova File "/usr/lib/
2023-07-13 06:14:08.622 1133261 ERROR nova found_version = temp_adapter.
2023-07-13 06:14:08.622 1133261 ERROR nova File "/usr/lib/
2023-07-13 06:14:08.622 1133261 ERROR nova return self.session.
2023-07-13 06:14:08.622 1133261 ERROR nova File "/usr/lib/
2023-07-13 06:14:08.622 1133261 ERROR nova return auth.get_
2023-07-13 06:14:08.622 1133261 ERROR nova File "/usr/lib/
2023-07-13 06:14:08.622 1133261 ERROR nova data = get_endpoint_
2023-07-13 06:14:08.622 1133261 ERROR nova File "/usr/lib/
2023-07-13 06:14:08.622 1133261 ERROR nova endpoint_data = service_
2023-07-13 06:14:08.622 1133261 ERROR nova File "/usr/lib/
2023-07-13 06:14:08.622 1133261 ERROR nova raise exceptions.
2023-07-13 06:14:08.622 1133261 ERROR nova keystoneauth1.
2023-07-13 06:36:36.692 1474436 WARNING keystoneauth. identity. generic. base [req-7609443e- 0609-4428- 97f8-6f5b41fc00 d4 - - - - -] Failed to discover available identity versions when contacting http:// keystone- internal. silo4.lab1. solutionsqa: 35357. Attempting to parse version from URL exceptions. http.BadRequest : Bad Request (HTTP 400) 0609-4428- 97f8-6f5b41fc00 d4 - - - - -] Unhandled error: keystoneauth1. exceptions. discovery. DiscoveryFailur e: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is c python3/ dist-packages/ keystoneauth1/ identity/ generic/ base.py" , line 136, in _do_create_plugin discovery( session, python3/ dist-packages/ keystoneauth1/ identity/ base.py" , line 608, in get_discovery get_discovery( session= session, url=url, python3/ dist-packages/ keystoneauth1/ discover. py", line 1452, in get_discovery authenticated) python3/ dist-packages/ keystoneauth1/ discover. py", line 535, in __init__ data(session, url, python3/ dist-packages/ keystoneauth1/ discover. py", line 102, in get_version_data authenticated) python3/ dist-packages/ keystoneauth1/ session. py", line 1141, in get python3/ dist-packages/ keystoneauth1/ session. py", line 986, in request from_response( resp, method, url) exceptions. http.BadRequest : Bad Request (HTTP 400)
.: keystoneauth1.
2023-07-13 06:36:36.696 1474436 CRITICAL nova [req-7609443e-
orrect. Bad Request (HTTP 400)
2023-07-13 06:36:36.696 1474436 ERROR nova Traceback (most recent call last):
2023-07-13 06:36:36.696 1474436 ERROR nova File "/usr/lib/
2023-07-13 06:36:36.696 1474436 ERROR nova disc = self.get_
2023-07-13 06:36:36.696 1474436 ERROR nova File "/usr/lib/
2023-07-13 06:36:36.696 1474436 ERROR nova return discover.
2023-07-13 06:36:36.696 1474436 ERROR nova File "/usr/lib/
2023-07-13 06:36:36.696 1474436 ERROR nova disc = Discover(session, url, authenticated=
2023-07-13 06:36:36.696 1474436 ERROR nova File "/usr/lib/
2023-07-13 06:36:36.696 1474436 ERROR nova self._data = get_version_
2023-07-13 06:36:36.696 1474436 ERROR nova File "/usr/lib/
2023-07-13 06:36:36.696 1474436 ERROR nova resp = session.get(url, headers=headers, authenticated=
2023-07-13 06:36:36.696 1474436 ERROR nova File "/usr/lib/
2023-07-13 06:36:36.696 1474436 ERROR nova return self.request(url, 'GET', **kwargs)
2023-07-13 06:36:36.696 1474436 ERROR nova File "/usr/lib/
2023-07-13 06:36:36.696 1474436 ERROR nova raise exceptions.
2023-07-13 06:36:36.696 1474436 ERROR nova keystoneauth1.