After Train migration we separate placement on another server checking the endpoint is correct
but when i perform a resize of instance get a error
keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://172.19.1.26:8778/resource_providers?in_tree=4abf4202-4da9-47e0-a5e5-70bbe5200265: HTTPConnectionPool(host='172.19.1.26', port=8778): Max retries exceeded with url:
Checking my endpoint URL on keystone show 172.19.1.26 as Nova API on port 8774
the problem is nova is mixing Nova IP with placement port the correct address should go is 172.19.1.22:8778
nova | compute | True | public | http://172.19.1.26:8774
placement | placement | True | public | http://172.19.1.22:8778
before they was together in stein release
as Workaround i use haproxy pointing to the placement ip and port
i add the full stack trace
Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 141, in _new_conn (self.host, self.port), self.timeout, **extra_kw) File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 83, in create_connection raise err File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 73, in create_connection sock.connect(sa) File "/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 253, in connect socket_checkerr(fd) File "/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 51, in socket_checkerr raise socket.error(err, errno.errorcode[err]) ConnectionRefusedError: [Errno 111] ECONNREFUSED During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 601, in urlopen chunked=chunked) File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 357, in _make_request conn.request(method, url, **httplib_request_kw) File "/usr/lib/python3.6/http/client.py", line 1264, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib/python3.6/http/client.py", line 1310, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib/python3.6/http/client.py", line 1259, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib/python3.6/http/client.py", line 1038, in _send_output self.send(msg) File "/usr/lib/python3.6/http/client.py", line 976, in send self.connect() File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 166, in connect conn = self._new_conn() File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 150, in _new_conn self, "Failed to establish a new connection: %s" % e) urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f9ad8efb4e0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/requests/adapters.py", line 440, in send timeout=timeout File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 639, in urlopen _stacktrace=sys.exc_info()[2]) File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 398, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='172.19.1.26', port=8778): Max retries exceeded with url: /resource_providers?in_tree=4abf4202-4da9-47e0-a5e5-70bbe5200265 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9ad8efb4e0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 979, in _send_request resp = self.session.request(method, url, **kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 520, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 630, in send r = adapter.send(request, **kwargs) File "/usr/lib/python3/dist-packages/requests/adapters.py", line 508, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='172.19.1.26', port=8778): Max retries exceeded with url: /resource_providers?in_tree=4abf4202-4da9-47e0-a5e5-70bbe5200265 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9ad8efb4e0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 995, in _send_request raise exceptions.ConnectFailure(msg) keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://172.19.1.26:8778/resource_providers?in_tree=4abf4202-4da9-47e0-a5e5-70bbe5200265: HTTPConnectionPool(host='172.19.1.26', port=8778): Max retries exceeded with url: /resource_providers?in_tree=4abf4202-4da9-47e0-a5e5-70bbe5200265 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9ad8efb4e0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 141, in _new_conn (self.host, self.port), self.timeout, **extra_kw) File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 83, in create_connection raise err File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 73, in create_connection sock.connect(sa) File "/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 253, in connect socket_checkerr(fd) File "/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 51, in socket_checkerr raise socket.error(err, errno.errorcode[err]) ConnectionRefusedError: [Errno 111] ECONNREFUSED During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 601, in urlopen chunked=chunked) File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 357, in _make_request conn.request(method, url, **httplib_request_kw) File "/usr/lib/python3.6/http/client.py", line 1264, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib/python3.6/http/client.py", line 1310, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib/python3.6/http/client.py", line 1259, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib/python3.6/http/client.py", line 1038, in _send_output self.send(msg) File "/usr/lib/python3.6/http/client.py", line 976, in send self.connect() File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 166, in connect conn = self._new_conn() File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 150, in _new_conn self, "Failed to establish a new connection: %s" % e) urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f9ad8f4df60>: Failed to establish a new connection: [Errno 111] ECONNREFUSED During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/requests/adapters.py", line 440, in send timeout=timeout File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 639, in urlopen _stacktrace=sys.exc_info()[2]) File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 398, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='172.19.1.26', port=8778): Max retries exceeded with url: /resource_providers?in_tree=4abf4202-4da9-47e0-a5e5-70bbe5200265 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9ad8f4df60>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 979, in _send_request resp = self.session.request(method, url, **kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 520, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 630, in send r = adapter.send(request, **kwargs) File "/usr/lib/python3/dist-packages/requests/adapters.py", line 508, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='172.19.1.26', port=8778): Max retries exceeded with url: /resource_providers?in_tree=4abf4202-4da9-47e0-a5e5-70bbe5200265 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9ad8f4df60>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 207, in decorated_function return function(self, context, *args, **kwargs) File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 4727, in prep_resize filter_properties, host_list) File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 4802, in _reschedule_resize_or_reraise six.reraise(*exc_info) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 4710, in prep_resize clean_shutdown) File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 4637, in _prep_resize limits=limits) as claim: File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 328, in inner return f(*args, **kwargs) File "/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py", line 255, in resize_claim limits=limits) File "/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py", line 378, in _move_claim self._update(elevated, cn) File "/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py", line 1237, in _update self._update_to_placement(context, compute_node, startup) File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/usr/lib/python3/dist-packages/retrying.py", line 206, in call return attempt.get(self._wrap_exception) File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/usr/lib/python3/dist-packages/nova/compute/resource_tracker.py", line 1151, in _update_to_placement context, compute_node.uuid, name=compute_node.hypervisor_hostname) File "/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 858, in get_provider_tree_and_ensure_root parent_provider_uuid=parent_provider_uuid) File "/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 640, in _ensure_resource_provider rps_to_refresh = self.get_providers_in_tree(context, uuid) File "/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 486, in get_providers_in_tree global_request_id=context.global_id) File "/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 237, in get global_request_id=global_request_id) File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 386, in get return self.request(url, 'GET', **kwargs) File "/usr/lib/python3/dist-packages/openstack/proxy.py", line 98, in request **kwargs) File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 248, in request return self.session.request(url, method, **kwargs) File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 888, in request resp = send(**kwargs) File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1018, in _send_request **kwargs) File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 995, in _send_request raise exceptions.ConnectFailure(msg) keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://172.19.1.26:8778/resource_providers?in_tree=4abf4202-4da9-47e0-a5e5-70bbe5200265: HTTPConnectionPool(host='172.19.1.26', port=8778): Max retries exceeded with url: /resource_providers?in_tree=4abf4202-4da9-47e0-a5e5-70bbe5200265 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9ad8f4df60>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))
Issue solved restarting nova-compute.