ci.centos - tripleo-quickstart-promote-ussuri-current-tripleo-delorean-minimal job is failing on overcloud deployment with Failed container(s): ['nova_wait_for_compute_service']
Two different kinds of issue observed:-
1) ERROR:nova_wait_for_compute_service:Error while waiting for nova-compute service to register
https://artifacts.ci.centos.org/rdo/jenkins-tripleo-quickstart-promote-ussuri-current-tripleo-delorean-minimal-111/undercloud/home/stack/overcloud_deploy.log
~~~
2020-10-29 19:29:17 | [ERROR]: Container(s) which finished with wrong return code:
2020-10-29 19:29:17 | ['nova_wait_for_compute_service']
2020-10-29 19:29:17 | [ERROR]: Container(s) which failed to be created by podman_container module:
2020-10-29 19:29:17 | ['nova_wait_for_compute_service']
2020-10-29 19:29:17 | 2020-10-29 19:29:17.052445 | 00322463-4992-bcd0-3177-000000006dac | FATAL | Check containers status | overcloud-novacompute-0 | error={"changed": false, "msg": "Failed container(s): ['nova_wait_for_compute_service'], check logs in /var/log/containers/stdouts/"}
2020-10-29 19:29:17 | 2020-10-29 19:29:17.055094 | 00322463-4992-bcd0-3177-000000006dac | TIMING | tripleo_container_manage : Check containers status | overcloud-novacompute-0 | 1:24:13.529097 | 2.70s
2020-10-29 19:29:17 |
2020-10-29 19:29:17 | PLAY RECAP *********************************************************************
2020-10-29 19:29:17 | overcloud-controller-0 : ok=632 changed=287 unreachable=0 failed=0 skipped=409 rescued=0 ignored=0
2020-10-29 19:29:17 | overcloud-novacompute-0 : ok=459 changed=186 unreachable=0 failed=1 skipped=382 rescued=0 ignored=0
~~~
https://artifacts.ci.centos.org/rdo/jenkins-tripleo-quickstart-promote-ussuri-current-tripleo-delorean-minimal-111/overcloud-novacompute-0/var/log/extra/podman/containers/nova_wait_for_compute_service/stdout.log
~~~
ERROR:nova_wait_for_compute_service:Error while waiting for nova-compute service to register
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
chunked=chunked,
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 421, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 416, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib64/python3.6/http/client.py", line 1346, in getresponse
response.begin()
File "/usr/lib64/python3.6/http/client.py", line 307, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python3.6/http/client.py", line 276, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 400, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/lib/python3.6/site-packages/urllib3/packages/six.py", line 702, in reraise
raise value.with_traceback(tb)
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
chunked=chunked,
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 421, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 416, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib64/python3.6/http/client.py", line 1346, in getresponse
response.begin()
File "/usr/lib64/python3.6/http/client.py", line 307, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python3.6/http/client.py", line 276, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1004, in _send_request
resp = self.session.request(method, url, **kwargs)
File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/container-config-scripts/nova_wait_for_compute_service.py", line 103, in <module>
service_list = nova.services.list(binary='nova-compute')
File "/usr/lib/python3.6/site-packages/novaclient/v2/services.py", line 53, in list
return self._list(url, "services")
File "/usr/lib/python3.6/site-packages/novaclient/base.py", line 253, in _list
resp, body = self.api.client.get(url)
File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 386, in get
return self.request(url, 'GET', **kwargs)
File "/usr/lib/python3.6/site-packages/novaclient/client.py", line 72, in request
**kwargs)
File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 545, in request
resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 248, in request
return self.session.request(url, method, **kwargs)
File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 913, in request
resp = send(**kwargs)
File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1020, in _send_request
raise exceptions.ConnectFailure(msg)
keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://172.16.2.236:8774/v2.1/os-services?binary=nova-compute: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
~~~
2) INFO:nova_wait_for_compute_service:Waiting for nova-compute service to register - as final log - "Nova-compute service registered" not noticed in logs
https://artifacts.ci.centos.org/rdo/jenkins-tripleo-quickstart-promote-ussuri-current-tripleo-delorean-minimal-110/overcloud-novacompute-0/var/log/extra/podman/containers/nova_wait_for_compute_service/stdout.log
~~~
INFO:nova_wait_for_compute_service:Waiting for nova-compute service to register
~~~
Hello,
Run on 09th Nov(build #113) passed
https:/
Closing this bug, for now, please feel free to reopen in case the issue reappears.