tripleo

Bug #1947403
Comment #9

Comment 9 for bug 1947403

Revision history for this message

Alex Schultz (alex-schultz) wrote on 2021-10-25:

The introspection api connection is going away and failing. We retry but it never succeeds. It should be noted even on a passing jobs, we get timeouts waiting for introspections but we don't see this api remote end closed connection.

https://logserver.rdoproject.org/64/814464/2/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/157ea66/logs/undercloud/var/log/containers/mistral/executor.log.txt.gz

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 400, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3.6/site-packages/urllib3/packages/six.py", line 702, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 421, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 416, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib64/python3.6/http/client.py", line 1365, in getresponse
    response.begin()
  File "/usr/lib64/python3.6/http/client.py", line 320, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python3.6/http/client.py", line 289, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 997, in _send_request
    resp = self.session.request(method, url, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/mistral/actions/openstack/base.py", line 117, in run
    result = method(**self._kwargs_for_run)
  File "/usr/lib/python3.6/site-packages/ironic_inspector_client/v1.py", line 245, in wait_for_finish
    status = self.get_status(node_id)
  File "/usr/lib/python3.6/site-packages/ironic_inspector_client/v1.py", line 210, in get_status
    return self.request('get', '/introspection/%s' % node_id).json()
  File "/usr/lib/python3.6/site-packages/ironic_inspector_client/common/http.py", line 195, in request
    raise_exc=False, **kwargs)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 906, in request
    resp = send(**kwargs)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1013, in _send_request
    raise exceptions.ConnectFailure(msg)
keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to https://192.168.24.2:13050/v1/introspection/c6cbbc32-804a-4a69-a31f-e3d0c52f8ff2: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
: keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to https://192.168.24.2:13050/v1/introspection/c6cbbc32-804a-4a69-a31f-e3d0c52f8ff2: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
2021-10-21 08:51:37.410 9 WARNING mistral.executors.default_executor [req-f69826c4-66cf-4489-afe5-9e986efbe688 d29623a86a3c4983ad8e63b903df24d3 e53fb58fa2c8434c91c8c1646fae0dcc - default default] The action raised an exception [action_ex_id=716d2717-6059-49b8-bece-742a6084ef13, msg='BaremetalIntrospectionAction.wait_for_finish failed: Unable to establish connection to https://192.168.24.2:13050/v1/introspection/c6cbbc32-804a-4a69-a31f-e3d0c52f8ff2: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))', action_cls='<class 'mistral.actions.action_factory.BaremetalIntrospectionAction'>', attributes='{'client_method_name': 'wait_for_finish'}', params='{'uuids': ['c6cbbc32-804a-4a69-a31f-e3d0c52f8ff2'], 'max_retries': 120, 'retry_interval': 10}']: mistral.exceptions.ActionException: BaremetalIntrospectionAction.wait_for_finish failed: Unable to establish connection to https://192.168.24.2:13050/v1/introspection/c6cbbc32-804a-4a69-a31f-e3d0c52f8ff2: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

The introspection api connection is going away and failing.  We retry but it never succeeds.  It should be noted even on a passing jobs, we get timeouts waiting for introspections but we don't see this api remote end closed connection.

https://logserver.rdoproject.org/64/814464/2/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/157ea66/logs/undercloud/var/log/containers/mistral/executor.log.txt.gz

2021-10-21 08:51:37.409 9 WARNING mistral.actions.openstack.base [req-f69826c4-66cf-4489-afe5-9e986efbe688 d29623a86a3c4983ad8e63b903df24d3 e53fb58fa2c8434c91c8c1646fae0dcc - default default] Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,                                                            
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 421, in _make_request
    six.raise_from(e, None)                                                     
  File "<string>", line 3, in raise_from                                        
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 416, in _make_request
    httplib_response = conn.getresponse()                                       
  File "/usr/lib64/python3.6/http/client.py", line 1365, in getresponse         
    response.begin()                                                            
  File "/usr/lib64/python3.6/http/client.py", line 320, in begin                
    version, status, reason = self._read_status()                               
  File "/usr/lib64/python3.6/http/client.py", line 289, in _read_status         
    raise RemoteDisconnected("Remote end closed connection without"             
http.client.RemoteDisconnected: Remote end closed connection without response   
                                                                                
During handling of the above exception, another exception occurred:             
                                                                                
Traceback (most recent call last):                                              
  File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout                                                             
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]             
  File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 400, in increment
    raise six.reraise(type(error), error, _stacktrace)                          
  File "/usr/lib/python3.6/site-packages/urllib3/packages/six.py", line 702, in reraise
    raise value.with_traceback(tb)                                              
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,                                                            
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 421, in _make_request
    six.raise_from(e, None)                                                     
  File "<string>", line 3, in raise_from                                        
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 416, in _make_request
    httplib_response = conn.getresponse()                                       
  File "/usr/lib64/python3.6/http/client.py", line 1365, in getresponse         
    response.begin()                                                            
  File "/usr/lib64/python3.6/http/client.py", line 320, in begin                
    version, status, reason = self._read_status()                               
  File "/usr/lib64/python3.6/http/client.py", line 289, in _read_status         
    raise RemoteDisconnected("Remote end closed connection without"             
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
                                                                                
During handling of the above exception, another exception occurred:             
                                                                                
Traceback (most recent call last):                                              
  File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 997, in _send_request
    resp = self.session.request(method, url, **kwargs)                          
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)                                       
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)                                         
  File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)                                 
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
                                                                                
During handling of the above exception, another exception occurred:             
                                                                                
Traceback (most recent call last):                                              
  File "/usr/lib/python3.6/site-packages/mistral/actions/openstack/base.py", line 117, in run
    result = method(**self._kwargs_for_run)                                     
  File "/usr/lib/python3.6/site-packages/ironic_inspector_client/v1.py", line 245, in wait_for_finish
    status = self.get_status(node_id)                                           
  File "/usr/lib/python3.6/site-packages/ironic_inspector_client/v1.py", line 210, in get_status
    return self.request('get', '/introspection/%s' % node_id).json()            
  File "/usr/lib/python3.6/site-packages/ironic_inspector_client/common/http.py", line 195, in request
    raise_exc=False, **kwargs)                                                  
  File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 906, in request
    resp = send(**kwargs)                                                       
  File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1013, in _send_request
    raise exceptions.ConnectFailure(msg)                                        
keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to https://192.168.24.2:13050/v1/introspection/c6cbbc32-804a-4a69-a31f-e3d0c52f8ff2: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
: keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to https://192.168.24.2:13050/v1/introspection/c6cbbc32-804a-4a69-a31f-e3d0c52f8ff2: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
2021-10-21 08:51:37.410 9 WARNING mistral.executors.default_executor [req-f69826c4-66cf-4489-afe5-9e986efbe688 d29623a86a3c4983ad8e63b903df24d3 e53fb58fa2c8434c91c8c1646fae0dcc - default default] The action raised an exception [action_ex_id=716d2717-6059-49b8-bece-742a6084ef13, msg='BaremetalIntrospectionAction.wait_for_finish failed: Unable to establish connection to https://192.168.24.2:13050/v1/introspection/c6cbbc32-804a-4a69-a31f-e3d0c52f8ff2: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))', action_cls='<class 'mistral.actions.action_factory.BaremetalIntrospectionAction'>', attributes='{'client_method_name': 'wait_for_finish'}', params='{'uuids': ['c6cbbc32-804a-4a69-a31f-e3d0c52f8ff2'], 'max_retries': 120, 'retry_interval': 10}']: mistral.exceptions.ActionException: BaremetalIntrospectionAction.wait_for_finish failed: Unable to establish connection to https://192.168.24.2:13050/v1/introspection/c6cbbc32-804a-4a69-a31f-e3d0c52f8ff2: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))