nova compute fails to create instance after nova compute reconfiguration with latest kolla-ansible images

[Debasis]

After reconfiguration of the OpenStack multinode deployment, I see the following logs in the nova-compute.log file on all the nova-compute nodes....

2023-08-25 14:51:17.283 7 ERROR nova.servicegroup.drivers.db [-] The services table record for the nova-compute service on host compute1 is gone. You either need to stop this service if it should be deleted or restart it to recreate the record in the database.: nova.exception_Remote.ServiceNotFound_Remote: Service 8 could not be found.
2023-08-25 14:51:27.296 7 INFO nova.servicegroup.drivers.db [-] Recovered from being unable to report status.
2023-08-25 14:51:37.289 7 ERROR nova.servicegroup.drivers.db [-] The services table record for the nova-compute service on host compute1 is gone. You either need to stop this service if it should be deleted or restart it to recreate the record in the database.: nova.exception_Remote.ServiceNotFound_Remote: Service 8 could not be found.
2023-08-25 14:51:47.297 7 INFO nova.servicegroup.drivers.db [-] Recovered from being unable to report status.
2023-08-25 14:51:56.576 7 INFO nova.compute.resource_tracker [None req-29662cae-63a6-4301-9617-9bd833d92005 - - - - - -] Instance ca7c3667-293e-4677-9f18-eed63f4592ac has allocations against this compute host but is not found in the database.
2023-08-25 14:51:56.603 7 INFO nova.compute.resource_tracker [None req-29662cae-63a6-4301-9617-9bd833d92005 - - - - - -] Instance 657e1c28-162a-4017-b27b-b0fe1ddaa02f has allocations against this compute host but is not found in the database.
2023-08-25 14:51:56.627 7 INFO nova.compute.resource_tracker [None req-29662cae-63a6-4301-9617-9bd833d92005 - - - - - -] Instance 6db232a9-690e-4895-999c-7a3ca38abbfb has allocations against this compute host but is not found in the database.
2023-08-25 14:51:56.657 7 INFO nova.compute.resource_tracker [None req-29662cae-63a6-4301-9617-9bd833d92005 - - - - - -] Instance 9e63207f-0c80-43b2-96e5-5a1650437e95 has allocations against this compute host but is not found in the database.
2023-08-25 14:51:56.693 7 INFO nova.compute.resource_tracker [None req-29662cae-63a6-4301-9617-9bd833d92005 - - - - - -] Instance f23ab3b6-1a75-4d8c-8b76-d5334825d06a has allocations against this compute host but is not found in the database.
2023-08-25 14:51:57.620 7 ERROR nova.compute.resource_tracker [None req-29662cae-63a6-4301-9617-9bd833d92005 - - - - - -] Unable to find services table record for nova-compute host compute1: nova.exception_Remote.ComputeHostNotFound_Remote: Compute host compute1 could not be found.
2023-08-25 14:51:57.658 7 WARNING nova.compute.manager [None req-29662cae-63a6-4301-9617-9bd833d92005 - - - - - -] Compute node 'compute1' not found in update_available_resource.: nova.exception_Remote.ComputeHostNotFound_Remote: Compute host 372 could not be found.
2023-08-25 14:52:07.289 7 ERROR nova.servicegroup.drivers.db [-] The services table record for the nova-compute service on host compute1 is gone. You either need to stop this service if it should be deleted or restart it to recreate the record in the database.: nova.exception_Remote.ServiceNotFound_Remote: Service 8 could not be found.
2023-08-25 14:52:17.292 7 INFO nova.servicegroup.drivers.db [-] Recovered from being unable to report status.
2023-08-25 14:52:27.291 7 ERROR nova.servicegroup.drivers.db [-] The services table record for the nova-compute service on host compute1 is gone. You either need to stop this service if it should be deleted or restart it to recreate the record in the database.: nova.exception_Remote.ServiceNotFound_Remote: Service 8 could not be found.
2023-08-25 14:52:37.306 7 INFO nova.servicegroup.drivers.db [-] Recovered from being unable to report status.
2023-08-25 14:52:47.300 7 ERROR nova.servicegroup.drivers.db [-] The services table record for the nova-compute service on host compute1 is gone. You either need to stop this service if it should be deleted or restart it to recreate the record in the database.: nova.exception_Remote.ServiceNotFound_Remote: Service 8 could not be found.
2023-08-25 14:52:49.211 7 INFO nova.network.neutron [None req-29662cae-63a6-4301-9617-9bd833d92005 - - - - - -] [instance: 523122ad-349c-44cb-abbe-b88c884e63e0] Failed storing info cache due to: Instance 523122ad-349c-44cb-abbe-b88c884e63e0 could not be found.. The instance may have moved to another cell during a cross-cell migration
2023-08-25 14:52:58.148 7 INFO nova.servicegroup.drivers.db [-] Recovered from being unable to report status.
2023-08-25 14:52:58.433 7 INFO nova.compute.resource_tracker [None req-29662cae-63a6-4301-9617-9bd833d92005 - - - - - -] Instance ca7c3667-293e-4677-9f18-eed63f4592ac has allocations against this compute host but is not found in the database.
2023-08-25 14:52:58.458 7 INFO nova.compute.resource_tracker [None req-29662cae-63a6-4301-9617-9bd833d92005 - - - - - -] Instance 657e1c28-162a-4017-b27b-b0fe1ddaa02f has allocations against this compute host but is not found in the database.
2023-08-25 14:52:58.487 7 INFO nova.compute.resource_tracker [None req-29662cae-63a6-4301-9617-9bd833d92005 - - - - - -] Instance 6db232a9-690e-4895-999c-7a3ca38abbfb has allocations against this compute host but is not found in the database.
2023-08-25 14:52:58.507 7 INFO nova.compute.resource_tracker [None req-29662cae-63a6-4301-9617-9bd833d92005 - - - - - -] Instance 9e63207f-0c80-43b2-96e5-5a1650437e95 has allocations against this compute host but is not found in the database.
2023-08-25 14:52:58.538 7 INFO nova.compute.resource_tracker [None req-29662cae-63a6-4301-9617-9bd833d92005 - - - - - -] Instance f23ab3b6-1a75-4d8c-8b76-d5334825d06a has allocations against this compute host but is not found in the database.
2023-08-25 14:53:08.128 7 ERROR nova.servicegroup.drivers.db [-] The services table record for the nova-compute service on host compute1 is gone. You either need to stop this service if it should be deleted or restart it to recreate the record in the database.: nova.exception_Remote.ServiceNotFound_Remote: Service 8 could not be found.
2023-08-25 14:53:18.131 7 INFO nova.servicegroup.drivers.db [-] Recovered from being unable to report status.
2023-08-25 14:53:28.132 7 ERROR nova.servicegroup.drivers.db [-] The services table record for the nova-compute service on host compute1 is gone. You either need to stop this service if it should be deleted or restart it to recreate the record in the database.: nova.exception_Remote.ServiceNotFound_Remote: Service 8 could not be found.

Any suggestion on how to temporarily resolve the issue will be appreciated.

[Przemysław Kuczyński (przemekkuczynski)]

Maybe this will help https://techglimpse.com/openstack-compute-error-no-host/
On what task kolla-ansible reconfigure fails ? Can You paste
grep ^[^#] /etc/kolla/globals.yml
and nova conf ?

[Debasis (debamondal)]

Sorry! I guess, I've understood the issue. It's related to the wrong tls configuration. Please ignore.

[Debasis (debamondal)]

Hi, No it doesn't fail. It reconfigures without any errors. All services are also up and enabled but when I try to create an instance it remains in the build state.

I thought, it's due to some tls issue but now I release that it's not. I'll post the globals.yml file once I'm back in office. But it was a perfectly working openstack cluster. I even could create instance, volume etc.

[Debasis (debamondal)]

Here's my nova.conf https://paste.opendev.org/show/bgEYjyzrq9DRWnMKmZ9n/ and globals.yml is here https://paste.opendev.org/show/b6MKuQHeW136WibRmdjg/