| 2019-04-25 13:34:48 |
Liam Young |
description |
If the url of the placement api changes after nova-compute has been started then placement updates fail as nova-compute appears to cache the old endpoint url. Eventually the nova scheduler will mark the node as down as it has sent no resource updates.
To reproduce, update the placement endpoint to something incorrect in keystone and restart nova-compute. Errors contacting the placement api will be reported every minute or so. Now, correct the entry in keystone. The errors will continue despite the catalogue now being correct. Restarting nova-compute fixes the issue.
In my deployment this occurred when the placement end point switched from http to https after the nova-compute node had started. This resulted in the following in the nova-compute log:
2019-04-25 09:58:12.175 31793 ERROR nova.scheduler.client.report [req-18b4f522-e702-4ee1-ba85-e565c8e9ac1e - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID 4f7c6844-d3b8-4710-be2c-8691a93fb58b. Got 400: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
Reason: You're speaking plain HTTP to an SSL-enabled server port.<br />
Instead use the HTTPS scheme to access this URL, please.<br />
</p>
<hr>
<address>Apache/2.4.29 (Ubuntu) Server at 10.5.0.36 Port 443</address>
</body></html>
.
2019-04-25 09:58:12.176 31793 DEBUG oslo_concurrency.lockutils [req-18b4f522-e702-4ee1-ba85-e565c8e9ac1e - - - - -] Lock "compute_resources" released by "nova.compute.resource_tracker.ResourceTracker._update_available_resource" :: held 0.099s inner /usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py:285
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager [req-18b4f522-e702-4ee1-ba85-e565c8e9ac1e - - - - -] Error updating resources for node juju-7a9f5c-zaza-19a393f3689b-16.project.serverstack.: nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 4f7c6844-d3b8-4710-be2c-8691a93fb58b
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager Traceback (most recent call last):
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 7778, in _update_available_resource_for_node
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager rt.update_available_resource(context, nodename)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py", line 721, in update_available_resource
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager self._update_available_resource(context, resources)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 274, in inner
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager return f(*args, **kwargs)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py", line 798, in _update_available_resource
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager self._update(context, cn)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager return Retrying(*dargs, **dkw).call(f, *args, **kw)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/retrying.py", line 206, in call
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager return attempt.get(self._wrap_exception)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager six.reraise(self.value[0], self.value[1], self.value[2])
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager raise value
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py", line 960, in _update
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager self._update_to_placement(context, compute_node)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py", line 912, in _update_to_placement
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager context, compute_node.uuid, name=compute_node.hypervisor_hostname)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/scheduler/client/__init__.py", line 35, in __run_method
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager return getattr(self.instance, __name)(*args, **kwargs)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 1006, in get_provider_tree_and_ensure_root
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 668, in _ensure_resource_provider
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager rps_to_refresh = self._get_providers_in_tree(context, uuid)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 74, in wrapper
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager return f(self, *a, **k)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 535, in _get_providers_in_tree
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager raise exception.ResourceProviderRetrievalFailed(uuid=uuid)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 4f7c6844-d3b8-4710-be2c-8691a93fb58b
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager |
If the url of the placement api changes after nova-compute has been started then placement updates fail as nova-compute appears to cache the old endpoint url.
To reproduce, update the placement endpoint to something incorrect in keystone and restart nova-compute. Errors contacting the placement api will be reported every minute or so. Now, correct the entry in keystone. The errors will continue despite the catalogue now being correct. Restarting nova-compute fixes the issue.
In my deployment this occurred when the placement end point switched from http to https after the nova-compute node had started. This resulted in the following in the nova-compute log:
2019-04-25 09:58:12.175 31793 ERROR nova.scheduler.client.report [req-18b4f522-e702-4ee1-ba85-e565c8e9ac1e - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID 4f7c6844-d3b8-4710-be2c-8691a93fb58b. Got 400: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
Reason: You're speaking plain HTTP to an SSL-enabled server port.<br />
Instead use the HTTPS scheme to access this URL, please.<br />
</p>
<hr>
<address>Apache/2.4.29 (Ubuntu) Server at 10.5.0.36 Port 443</address>
</body></html>
.
2019-04-25 09:58:12.176 31793 DEBUG oslo_concurrency.lockutils [req-18b4f522-e702-4ee1-ba85-e565c8e9ac1e - - - - -] Lock "compute_resources" released by "nova.compute.resource_tracker.ResourceTracker._update_available_resource" :: held 0.099s inner /usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py:285
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager [req-18b4f522-e702-4ee1-ba85-e565c8e9ac1e - - - - -] Error updating resources for node juju-7a9f5c-zaza-19a393f3689b-16.project.serverstack.: nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 4f7c6844-d3b8-4710-be2c-8691a93fb58b
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager Traceback (most recent call last):
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 7778, in _update_available_resource_for_node
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager rt.update_available_resource(context, nodename)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py", line 721, in update_available_resource
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager self._update_available_resource(context, resources)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 274, in inner
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager return f(*args, **kwargs)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py", line 798, in _update_available_resource
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager self._update(context, cn)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager return Retrying(*dargs, **dkw).call(f, *args, **kw)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/retrying.py", line 206, in call
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager return attempt.get(self._wrap_exception)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager six.reraise(self.value[0], self.value[1], self.value[2])
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager raise value
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py", line 960, in _update
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager self._update_to_placement(context, compute_node)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py", line 912, in _update_to_placement
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager context, compute_node.uuid, name=compute_node.hypervisor_hostname)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/scheduler/client/__init__.py", line 35, in __run_method
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager return getattr(self.instance, __name)(*args, **kwargs)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 1006, in get_provider_tree_and_ensure_root
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 668, in _ensure_resource_provider
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager rps_to_refresh = self._get_providers_in_tree(context, uuid)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 74, in wrapper
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager return f(self, *a, **k)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager File "/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 535, in _get_providers_in_tree
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager raise exception.ResourceProviderRetrievalFailed(uuid=uuid)
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 4f7c6844-d3b8-4710-be2c-8691a93fb58b
2019-04-25 09:58:12.177 31793 ERROR nova.compute.manager |
|
