Collect logs in OVB job fails to fetch overcloud nodes and their logs

[Gabriele Cerami]

In recent OVB jobs no logs from overcloud nodes are available.

Logs in one of these jobs e.g. at:

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky/204561b/logs/quickstart_collect_logs.log

show

TASK [tripleo-inventory : fetch overcloud node names and IPs] ******************
task path: /home/zuul/src/git.openstack.org/openstack/tripleo-quickstart/roles/tripleo-inventory/tasks/inventory.yml:13
Thursday 15 November 2018 09:13:09 +0000 (0:00:02.365) 0:00:44.107 *****
fatal: [undercloud]: FAILED! => {
    "changed": true,
    "cmd": "source /home/zuul/stackrc; python /home/zuul/get-overcloud-nodes.py",
    "delta": "0:00:00.747331",
    "end": "2018-11-15 09:13:10.578652",
    "rc": 1,
    "start": "2018-11-15 09:13:09.831321"
}

STDERR:

No handlers could be found for logger "keystoneauth.identity.generic.base"
Traceback (most recent call last):
  File "/home/zuul/get-overcloud-nodes.py", line 50, in <module>
    for server in nova.servers.list()
  File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 858, in list
    "servers")
  File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 257, in _list
    resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 328, in get
    return self.request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 77, in request
    **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 487, in request
    resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 213, in request
    return self.session.request(url, method, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 684, in request
    auth_headers = self.get_auth_headers(auth)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 1071, in get_auth_headers
    return auth.get_headers(self, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/plugin.py", line 95, in get_headers
    token = self.get_token(session)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 88, in get_token
    return self.get_access(session).auth_token
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
    self.auth_ref = self.get_auth_ref(session)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref
    return self._plugin.get_auth_ref(session, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/v3/base.py", line 178, in get_auth_ref
    authenticated=False, log=False, **rkwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 1019, in post
    return self.request(url, 'POST', **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 814, in request
    resp = send(**kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 907, in _send_request
    raise exceptions.SSLError(msg)
keystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://192.168.24.2:13000/v3/auth/tokens: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)

Collect logs in unable to connect to keystone, fetch the node informations and then the logs.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/618205

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alex Schultz (<email address hidden>) on branch: master
Review: https://review.openstack.org/618205
Reason: Clearing the gate. Do not retore this until being given the all clear. See http://lists.openstack.org/pipermail/openstack-discuss/2018-November/000368.html

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/618205
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart/commit/?id=87384e5de4995e3a4e23dd01f6c0197e1f489669
Submitter: Zuul
Branch: master

commit 87384e5de4995e3a4e23dd01f6c0197e1f489669
Author: Gabriele Cerami <email address hidden>
Date: Thu Nov 15 16:08:00 2018 +0000

    Collect logs: disable SSL certificate verification

    OVB jobs are currently failing to collect logs from the overcloud nodes.
    The script does not disable verification, and we fail to gather nodes
    information.
    This patch restructure the client call to use keystone tools and disable
    verification

    Change-Id: I762c76c7713e7603e357b9878318a9723a6c2d72
    Closes-Bug: #1803561

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-quickstart 2.1.1 release.