If the url of the placement api changes after nova-compute has been started then placement updates fail as nova-compute appears to cache the old endpoint url. Eventually the nova scheduler will mark the node as down as it has sent no resource updates.
To reproduce, update the placement endpoint to something incorrect in keystone and restart nova-compute. Errors contacting the placement api will be reported every minute or so. Now, correct the entry in keystone. The errors will continue despite the catalogue now being correct. Restarting nova-compute fixes the issue.
In my deployment this occurred when the placement end point switched from http to https after the nova-compute node had started. This resulted in the following in the nova-compute log:
2019-04-25 09:58:12.175 31793 ERROR nova.scheduler.client.report [req-18b4f522-e702-4ee1-ba85-e565c8e9ac1e - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID 4f7c6844-d3b8-4710-be2c-8691a93fb58b. Got 400: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
Reason: You're speaking plain HTTP to an SSL-enabled server port.<br />
Instead use the HTTPS scheme to access this URL, please.<br />
</p>
<hr>
<address>Apache/2.4.29 (Ubuntu) Server at 10.5.0.36 Port 443</address>
</body></html>
.
2019-04-25 09:58:12.176 31793 DEBUG oslo_concurrency.lockutils [req-18b4f522-e702-4ee1-ba85-e565c8e9ac1e - - - - -] Lock "compute_resources" released by "nova.compute.resource_tracker.ResourceTracker._update_available_resource" :: held 0.099s inner /usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py:285
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager [req-18b4f522-e702-4ee1-ba85-e565c8e9ac1e - - - - -] Error updating resources for node juju-7a9f5c-zaza-19a393f3689b-16.project.serverstack.: nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 4f7c6844-d3b8-4710-be2c-8691a93fb58b
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager Traceback (most recent call last):
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 7778, in _update_available_resource_for_node
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager rt.update_available_resource(context, nodename)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py", line 721, in update_available_resource
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager self._update_available_resource(context, resources)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 274, in inner
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager return f(*args, **kwargs)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py", line 798, in _update_available_resource
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager self._update(context, cn)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager return Retrying(*dargs, **dkw).call(f, *args, **kw)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/retrying.py", line 206, in call
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager return attempt.get(self._wrap_exception)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager six.reraise(self.value[0], self.value[1], self.value[2])
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager raise value
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py", line 960, in _update
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager self._update_to_placement(context, compute_node)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py", line 912, in _update_to_placement
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager context, compute_node.uuid, name=compute_node.hypervisor_hostname)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/scheduler/client/__init__.py", line 35, in __run_method
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager return getattr(self.instance, __name)(*args, **kwargs)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 1006, in get_provider_tree_and_ensure_root
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 668, in _ensure_resource_provider
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager rps_to_refresh = self._get_providers_in_tree(context, uuid)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 74, in wrapper
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager return f(self, *a, **k)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 535, in _get_providers_in_tree
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager raise exception.ResourceProviderRetrievalFailed(uuid=uuid)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 4f7c6844-d3b8-4710-be2c-8691a93fb58b
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager
If the url of the placement api changes after nova-compute has been started then placement updates fail as nova-compute appears to cache the old endpoint url. Eventually the nova scheduler will mark the node as down as it has sent no resource updates.
To reproduce, update the placement endpoint to something incorrect in keystone and restart nova-compute. Errors contacting the placement api will be reported every minute or so. Now, correct the entry in keystone. The errors will continue despite the catalogue now being correct. Restarting nova-compute fixes the issue.
In my deployment this occurred when the placement end point switched from http to https after the nova-compute node had started. This resulted in the following in the nova-compute log:
2019-04-25 09:58:12.175 31793 ERROR nova.scheduler. client. report [req-18b4f522- e702-4ee1- ba85-e565c8e9ac 1e - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID 4f7c6844- d3b8-4710- be2c-8691a93fb5 8b. Got 400: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> Apache/ 2.4.29 (Ubuntu) Server at 10.5.0.36 Port 443</address> y.lockutils [req-18b4f522- e702-4ee1- ba85-e565c8e9ac 1e - - - - -] Lock "compute_resources" released by "nova.compute. resource_ tracker. ResourceTracker ._update_ available_ resource" :: held 0.099s inner /usr/lib/ python3/ dist-packages/ oslo_concurrenc y/lockutils. py:285 manager [req-18b4f522- e702-4ee1- ba85-e565c8e9ac 1e - - - - -] Error updating resources for node juju-7a9f5c- zaza-19a393f368 9b-16.project. serverstack. : nova.exception. ResourceProvide rRetrievalFaile d: Failed to get resource provider with UUID 4f7c6844- d3b8-4710- be2c-8691a93fb5 8b manager Traceback (most recent call last): manager File "/usr/lib/ python3/ dist-packages/ nova/compute/ manager. py", line 7778, in _update_ available_ resource_ for_node manager rt.update_ available_ resource( context, nodename) manager File "/usr/lib/ python3/ dist-packages/ nova/compute/ resource_ tracker. py", line 721, in update_ available_ resource manager self._update_ available_ resource( context, resources) manager File "/usr/lib/ python3/ dist-packages/ oslo_concurrenc y/lockutils. py", line 274, in inner manager return f(*args, **kwargs) manager File "/usr/lib/ python3/ dist-packages/ nova/compute/ resource_ tracker. py", line 798, in _update_ available_ resource manager self._update( context, cn) manager File "/usr/lib/ python3/ dist-packages/ retrying. py", line 49, in wrapped_f manager return Retrying(*dargs, **dkw).call(f, *args, **kw) manager File "/usr/lib/ python3/ dist-packages/ retrying. py", line 206, in call manager return attempt. get(self. _wrap_exception ) manager File "/usr/lib/ python3/ dist-packages/ retrying. py", line 247, in get manager six.reraise( self.value[ 0], self.value[1], self.value[2]) manager File "/usr/lib/ python3/ dist-packages/ six.py" , line 693, in reraise manager raise value manager File "/usr/lib/ python3/ dist-packages/ retrying. py", line 200, in call manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False) manager File "/usr/lib/ python3/ dist-packages/ nova/compute/ resource_ tracker. py", line 960, in _update manager self._update_ to_placement( context, compute_node) manager File "/usr/lib/ python3/ dist-packages/ nova/compute/ resource_ tracker. py", line 912, in _update_ to_placement manager context, compute_node.uuid, name=compute_ node.hypervisor _hostname) manager File "/usr/lib/ python3/ dist-packages/ nova/scheduler/ client/ __init_ _.py", line 35, in __run_method manager return getattr( self.instance, __name)(*args, **kwargs) manager File "/usr/lib/ python3/ dist-packages/ nova/scheduler/ client/ report. py", line 1006, in get_provider_ tree_and_ ensure_ root manager parent_ provider_ uuid=parent_ provider_ uuid) manager File "/usr/lib/ python3/ dist-packages/ nova/scheduler/ client/ report. py", line 668, in _ensure_ resource_ provider manager rps_to_refresh = self._get_ providers_ in_tree( context, uuid) manager File "/usr/lib/ python3/ dist-packages/ nova/scheduler/ client/ report. py", line 74, in wrapper manager return f(self, *a, **k) manager File "/usr/lib/ python3/ dist-packages/ nova/scheduler/ client/ report. py", line 535, in _get_providers_ in_tree manager raise exception. ResourceProvide rRetrievalFaile d(uuid= uuid) manager nova.exception. ResourceProvide rRetrievalFaile d: Failed to get resource provider with UUID 4f7c6844- d3b8-4710- be2c-8691a93fb5 8b manager
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
Reason: You're speaking plain HTTP to an SSL-enabled server port.<br />
Instead use the HTTPS scheme to access this URL, please.<br />
</p>
<hr>
<address>
</body></html>
.
2019-04-25 09:58:12.176 31793 DEBUG oslo_concurrenc
2019-04-25 09:58:12.177 31793 ERROR nova.compute.
2019-04-25 09:58:12.177 31793 ERROR nova.compute.
2019-04-25 09:58:12.177 31793 ERROR nova.compute.
2019-04-25 09:58:12.177 31793 ERROR nova.compute.
2019-04-25 09:58:12.177 31793 ERROR nova.compute.
2019-04-25 09:58:12.177 31793 ERROR nova.compute.
2019-04-25 09:58:12.177 31793 ERROR nova.compute.
2019-04-25 09:58:12.177 31793 ERROR nova.compute.
2019-04-25 09:58:12.177 31793 ERROR nova.compute.
2019-04-25 09:58:12.177 31793 ERROR nova.compute.
2019-04-25 09:58:12.177 31793 ERROR nova.compute.
2019-04-25 09:58:12.177 31793 ERROR nova.compute.
2019-04-25 09:58:12.177 31793 ERROR nova.compute.
2019-04-25 09:58:12.177 31793 ERROR nova.compute.
2019-04-25 09:58:12.177 31793 ERROR nova.compute.
2019-04-25 09:58:12.177 31793 ERROR nova.compute.
2019-04-25 09:58:12.177 31793 ERROR nova.compute.
2019-04-25 09:58:12.177 31793 ERROR nova.compute.
2019-04-25 09:58:12.177 31793 ERROR nova.compute.
2019-04-25 09:58:12.177 31793 ERROR nova.compute.
2019-04-25 09:58:12.177 31793 ERROR nova.compute.
2019-04-25 09:58:12.177 31793 ERROR nova.compute.
2019-04-25 09:58:12.177 31793 ERROR nova.compute.
2019-04-25 09:58:12.177 31793 ERROR nova.compute.
2019-04-25 09:58:12.177 31793 ERROR nova.compute.
2019-04-25 09:58:12.177 31793 ERROR nova.compute.
2019-04-25 09:58:12.177 31793 ERROR nova.compute.
2019-04-25 09:58:12.177 31793 ERROR nova.compute.
2019-04-25 09:58:12.177 31793 ERROR nova.compute.
2019-04-25 09:58:12.177 31793 ERROR nova.compute.
2019-04-25 09:58:12.177 31793 ERROR nova.compute.
2019-04-25 09:58:12.177 31793 ERROR nova.compute.
2019-04-25 09:58:12.177 31793 ERROR nova.compute.
2019-04-25 09:58:12.177 31793 ERROR nova.compute.
2019-04-25 09:58:12.177 31793 ERROR nova.compute.
2019-04-25 09:58:12.177 31793 ERROR nova.compute.