The placement service for 192.168.24.100:regionOne exists but does not have any supported versions.

[Jiri Podivin]

Nova Conductor container running under featureset 010 fails with:

"The placement service for 192.168.24.100:regionOne exists but does not have any supported versions."

The nova-conductor fails a health check slightly before that, but I'm not certain about connection between the two events.

As of now I don't know what is the cause of the error or how it is related to the featureset and nodeset used.

Release: master

Trace:
------

2021-10-29 16:37:54.218 9 CRITICAL nova [-] Unhandled error: openstack.exceptions.NotSupported: The placement service for 192.168.24.100:regionOne exists but does not have any supported versions.
2021-10-29 16:37:54.218 9 ERROR nova Traceback (most recent call last):
2021-10-29 16:37:54.218 9 ERROR nova File "/usr/bin/nova-conductor", line 10, in <module>
2021-10-29 16:37:54.218 9 ERROR nova sys.exit(main())
2021-10-29 16:37:54.218 9 ERROR nova File "/usr/lib/python3.6/site-packages/nova/cmd/conductor.py", line 46, in main
2021-10-29 16:37:54.218 9 ERROR nova topic=rpcapi.RPC_TOPIC)
2021-10-29 16:37:54.218 9 ERROR nova File "/usr/lib/python3.6/site-packages/nova/service.py", line 256, in create
2021-10-29 16:37:54.218 9 ERROR nova periodic_interval_max=periodic_interval_max)
2021-10-29 16:37:54.218 9 ERROR nova File "/usr/lib/python3.6/site-packages/nova/service.py", line 116, in __init__
2021-10-29 16:37:54.218 9 ERROR nova self.manager = manager_class(host=self.host, *args, **kwargs)
2021-10-29 16:37:54.218 9 ERROR nova File "/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 119, in __init__
2021-10-29 16:37:54.218 9 ERROR nova self.compute_task_mgr = ComputeTaskManager()
2021-10-29 16:37:54.218 9 ERROR nova File "/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 244, in __init__
2021-10-29 16:37:54.218 9 ERROR nova self.report_client = report.SchedulerReportClient()
2021-10-29 16:37:54.218 9 ERROR nova File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 187, in __init__
2021-10-29 16:37:54.218 9 ERROR nova self._client = self._create_client()
2021-10-29 16:37:54.218 9 ERROR nova File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 230, in _create_client
2021-10-29 16:37:54.218 9 ERROR nova client = self._adapter or utils.get_sdk_adapter('placement')
2021-10-29 16:37:54.218 9 ERROR nova File "/usr/lib/python3.6/site-packages/nova/utils.py", line 983, in get_sdk_adapter
2021-10-29 16:37:54.218 9 ERROR nova return getattr(conn, service_type)
2021-10-29 16:37:54.218 9 ERROR nova File "/usr/lib/python3.6/site-packages/openstack/service_description.py", line 87, in __get__
2021-10-29 16:37:54.218 9 ERROR nova proxy = self._make_proxy(instance)
2021-10-29 16:37:54.218 9 ERROR nova File "/usr/lib/python3.6/site-packages/openstack/service_description.py", line 271, in _make_proxy
2021-10-29 16:37:54.218 9 ERROR nova region_name=region_name))
2021-10-29 16:37:54.218 9 ERROR nova openstack.exceptions.NotSupported: The placement service for 192.168.24.100:regionOne exists but does not have any supported versions.
2021-10-29 16:37:54.218 9 ERROR nova

Logs:
-----
https://logserver.rdoproject.org/99/36499/6/check/periodic-tripleo-ci-centos-8-containers-multinode-compute-master-validation/91262ed/logs/subnode-1/var/log/containers/nova/nova-conductor.log.txt.gz

https://logserver.rdoproject.org/99/36499/6/check/periodic-tripleo-ci-centos-8-containers-multinode-compute-master-validation/91262ed/logs/subnode-1/var/log/extra/failed_services.txt.gz

Full logs for the job: https://logserver.rdoproject.org/99/36499/6/check/periodic-tripleo-ci-centos-8-containers-multinode-compute-master-validation/91262ed/

[Balazs Gibizer (balazs-gibizer)]

Is this still happening? I do remember an Apache change that caused a difference in trailing / handling in the proxy configuration [1]. I feel this might be related.

[1] https://review.opendev.org/c/openstack/devstack/+/811303

[Ronelle Landy (rlandy)]

https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-standalone-on-multinode-ipa&skip=0

does not reproduce on check:

https://ef2ad358dccb56dd5b1d-616db0a32cd5a8455c371f4b45f3133e.ssl.cf5.rackcdn.com/828144/4/check/tripleo-ci-centos-9-standalone-on-multinode-ipa/e5d5640/logs/undercloud/var/log/tempest/stestr_results.html

[Ronelle Landy (rlandy)]

sorry - posted on wrong bug above

[Mario Chirinos (mario-chirinos)]

I have the same problem, did you find a solution?

[ZhangLong (ankele)]

I met the same problem, and I find the placement database is empty, even after "placement-manage db sync". And I find my /etc/placement/placement.conf is empty. Configure it correctly, and restart placement-api wsgi service, I got success.

[Moises Emilio Benzan Mora (moisesbenzan)]

Hello, SQA team found this bug during our testing.

From running the tempest suite, we get the following error:
=================================================================================================================
 tempest.api.compute.servers.test_attach_interfaces.AttachInterfacesTestJSON.test_reassign_port_between_servers
fail 110.351
3b55a32e-4841-48b3-a3aa-050d261129c1

Traceback (most recent call last):
  File "/home/ubuntu/snap/fcbtest/50/.rally/verification/verifier-69ebebc9-78df-4ad7-9740-23b4f54ebb96/repo/tempest/api/compute/servers/test_attach_interfaces.py", line 337, in test_reassign_port_between_servers
    iface = self.interfaces_client.create_interface(
  File "/home/ubuntu/snap/fcbtest/50/.rally/verification/verifier-69ebebc9-78df-4ad7-9740-23b4f54ebb96/repo/tempest/lib/services/compute/interfaces_client.py", line 47, in create_interface
    resp, body = self.post('servers/%s/os-interface' % server_id,
  File "/home/ubuntu/snap/fcbtest/50/.rally/verification/verifier-69ebebc9-78df-4ad7-9740-23b4f54ebb96/repo/tempest/lib/common/rest_client.py", line 300, in post
    return self.request('POST', url, extra_headers, headers, body, chunked)
  File "/home/ubuntu/snap/fcbtest/50/.rally/verification/verifier-69ebebc9-78df-4ad7-9740-23b4f54ebb96/repo/tempest/lib/services/compute/base_compute_client.py", line 47, in request
    resp, resp_body = super(BaseComputeClient, self).request(
  File "/home/ubuntu/snap/fcbtest/50/.rally/verification/verifier-69ebebc9-78df-4ad7-9740-23b4f54ebb96/repo/tempest/lib/common/rest_client.py", line 742, in request
    self._error_checker(resp, resp_body)
  File "/home/ubuntu/snap/fcbtest/50/.rally/verification/verifier-69ebebc9-78df-4ad7-9740-23b4f54ebb96/repo/tempest/lib/common/rest_client.py", line 922, in _error_checker
    raise exceptions.ServerFault(resp_body, resp=resp,
tempest.lib.exceptions.ServerFault: Got server fault
Details: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'nova.exception_Remote.PortBindingFailed_Remote'>

=====================================================================================================================

Looking into the Nova API logs, we then get:
=====================================================================================================================
2023-05-30 22:27:37.882 395255 ERROR nova
2023-05-30 22:27:49.883 404005 INFO os_vif [-] Loaded VIF plugins: linux_bridge, noop, ovs
2023-05-30 22:28:00.147 404005 WARNING nova.conductor.api [req-01768358-b6bc-4503-82e6-2b74b5a8e7be - - - - -] Timed out waiting for nova-conductor. Is it running? Or did this service start before nova-
conductor? Reattempting establishment of nova-conductor connection...: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 4f27a62457e74f2783f79b86415da0f7
=====================================================================================================================

And taking a look at the nova-conductor logs, it shows that around that same time it was failing to initialize the placement client:
=====================================...