How to reproduce?
Deploying tripleo with tripleo-quickstart:
- ./quickstart.sh -R master-tripleo-ci --tags all -T none -p quickstart.yml 127.0.0.2
- ./quickstart.sh -R master-tripleo-ci --no-clone --tags all -I -T none -p quickstart-extras-undercloud.yml 127.0.0.2
- ./quickstart.sh -R master-tripleo-ci --no-clone --tags all -I -T none -p quickstart-extras-overcloud-prep.yml 127.0.0.2
It took a lot of time to fail.
Some logs from the undercloud:
##
## overcloud_introspect.log
######
2020-10-21 14:47:27 | File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 34, in run
2020-10-21 14:47:27 | super(Command, self).run(parsed_args)
2020-10-21 14:47:27 | File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run
2020-10-21 14:47:27 | return super(Command, self).run(parsed_args)
2020-10-21 14:47:27 | File "/usr/lib/python3.6/site-packages/cliff/command.py", line 187, in run
2020-10-21 14:47:27 | return_code = self.take_action(parsed_args) or 0
2020-10-21 14:47:27 | File "/usr/lib/python3.6/site-packages/tripleoclient/v2/overcloud_node.py", line 190, in take_action
2020-10-21 14:47:27 | verbosity=oooutils.playbook_verbosity(self=self)
2020-10-21 14:47:27 | File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/baremetal.py", line 237, in introspect_manageable_nodes
2020-10-21 14:47:27 | verbosity=verbosity
2020-10-21 14:47:27 | File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/baremetal.py", line 191, in introspect
2020-10-21 14:47:27 | "retry_timeout": retry_timeout,
2020-10-21 14:47:27 | File "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 722, in run_ansible_playbook
2020-10-21 14:47:27 | raise RuntimeError(err_msg)
2020-10-21 14:47:27 | RuntimeError: Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-baremetal-introspect.yaml, Run Status: failed, Return Code: 2
2020-10-21 14:47:27 | Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-baremetal-introspect.yaml, Run Status: failed, Return Code: 2
##
## part of the /var/log/containers/stdouts/neutron_api.log
######
2020-10-21T14:28:28.378780369+00:00 stderr F Removing descriptor: 14
2020-10-21T14:28:28.448102856+00:00 stderr F Traceback (most recent call last):
2020-10-21T14:28:28.448102856+00:00 stderr F File "/usr/lib/python3.6/site-packages/neutron_lib/placement/client.py", line 57, in wrapper
2020-10-21T14:28:28.448102856+00:00 stderr F return f(self, *a, **k)
2020-10-21T14:28:28.448102856+00:00 stderr F File "/usr/lib/python3.6/site-packages/neutron_lib/placement/client.py", line 554, in list_aggregates
2020-10-21T14:28:28.448102856+00:00 stderr F return self._get(url).json()
2020-10-21T14:28:28.448102856+00:00 stderr F File "/usr/lib/python3.6/site-packages/neutron_lib/placement/client.py", line 191, in _get
2020-10-21T14:28:28.448102856+00:00 stderr F **kwargs)
2020-10-21T14:28:28.448102856+00:00 stderr F File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1131, in get
2020-10-21T14:28:28.448102856+00:00 stderr F return self.request(url, 'GET', **kwargs)
2020-10-21T14:28:28.448102856+00:00 stderr F File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 780, in request
2020-10-21T14:28:28.448102856+00:00 stderr F auth_headers = self.get_auth_headers(auth)
2020-10-21T14:28:28.448102856+00:00 stderr F File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1191, in get_auth_headers
2020-10-21T14:28:28.448102856+00:00 stderr F return auth.get_headers(self, **kwargs)
2020-10-21T14:28:28.448102856+00:00 stderr F File "/usr/lib/python3.6/site-packages/keystoneauth1/plugin.py", line 95, in get_headers
2020-10-21T14:28:28.448102856+00:00 stderr F token = self.get_token(session)
2020-10-21T14:28:28.448102856+00:00 stderr F File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 88, in get_token
2020-10-21T14:28:28.448102856+00:00 stderr F return self.get_access(session).auth_token
2020-10-21T14:28:28.448102856+00:00 stderr F File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
2020-10-21T14:28:28.448102856+00:00 stderr F self.auth_ref = self.get_auth_ref(session)
2020-10-21T14:28:28.448102856+00:00 stderr F File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref
2020-10-21T14:28:28.448102856+00:00 stderr F return self._plugin.get_auth_ref(session, **kwargs)
2020-10-21T14:28:28.448102856+00:00 stderr F File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/v3/base.py", line 188, in get_auth_ref
2020-10-21T14:28:28.448102856+00:00 stderr F authenticated=False, log=False, **rkwargs)
2020-10-21T14:28:28.448102856+00:00 stderr F File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1139, in post
2020-10-21T14:28:28.448102856+00:00 stderr F return self.request(url, 'POST', **kwargs)
2020-10-21T14:28:28.448102856+00:00 stderr F File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 976, in request
2020-10-21T14:28:28.448102856+00:00 stderr F raise exceptions.from_response(resp, method, url)
2020-10-21T14:28:28.448102856+00:00 stderr F keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-2b0ddf8b-47cb-4fc7-9884-84cca37e3462)
2020-10-21T14:28:28.448102856+00:00 stderr F
##
## Errors of the req-2b0ddf8b-47cb-4fc7-9884-84cca37e3462
#####
undercloud) [stack@undercloud ~]$ sudo grep -R -i req-2b0ddf8b-47cb-4fc7-9884-84cca37e3462 /var/log/*
/var/log/containers/keystone/keystone.log:2020-10-21 14:28:28.357 158 DEBUG keystone.server.flask.request_processing.req_logging [req-2b0ddf8b-47cb-4fc7-9884-84cca37e3462 - - - - -] REQUEST_METHOD: `POST` log_request_info /usr/lib/python3.6/site-packages/keystone/server/flask/request_processing/req_logging.py:27
/var/log/containers/keystone/keystone.log:2020-10-21 14:28:28.357 158 DEBUG keystone.server.flask.request_processing.req_logging [req-2b0ddf8b-47cb-4fc7-9884-84cca37e3462 - - - - -] SCRIPT_NAME: `` log_request_info /usr/lib/python3.6/site-packages/keystone/server/flask/request_processing/req_logging.py:28
/var/log/containers/keystone/keystone.log:2020-10-21 14:28:28.357 158 DEBUG keystone.server.flask.request_processing.req_logging [req-2b0ddf8b-47cb-4fc7-9884-84cca37e3462 - - - - -] PATH_INFO: `/v3/auth/tokens` log_request_info /usr/lib/python3.6/site-packages/keystone/server/flask/request_processing/req_logging.py:29
/var/log/containers/keystone/keystone.log:2020-10-21 14:28:28.370 158 WARNING keystone.auth.plugins.core [req-2b0ddf8b-47cb-4fc7-9884-84cca37e3462 - - - - -] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova.
/var/log/containers/keystone/keystone.log:2020-10-21 14:28:28.444 158 WARNING keystone.server.flask.application [req-2b0ddf8b-47cb-4fc7-9884-84cca37e3462 - - - - -] Authorization failed. The request you have made requires authentication. from 192.168.24.1: keystone.exception.Unauthorized: The request you have made requires authentication.
/var/log/containers/stdouts/neutron_api.log:2020-10-21T14:28:28.448102856+00:00 stderr F keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-2b0ddf8b-47cb-4fc7-9884-84cca37e3462)
Checking the introspection there are several SElinux issues on the overcloud node console such as
https:/