periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-queens-upload job is failing to fetch overcloud node names

Bug #1775365 reported by Arx Cruz
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Arx Cruz

Bug Description

Logs:

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-queens-upload/45758d8/console.txt.gz

Failure:
2018-06-06 07:01:00.552 | fatal: [undercloud]: FAILED! => {"changed": true, "cmd": "source /home/jenkins/stackrc; python /home/jenkins/get-overcloud-nodes.py", "delta": "0:00:00.792770", "end": "2018-06-06 07:01:00.502989", "failed": true, "msg": "non-zero return code", "rc": 1, "start": "2018-06-06 07:00:59.710219", "stderr": "Traceback (most recent call last):\n File \"/home/jenkins/get-overcloud-nodes.py\", line 44, in <module>\n for server in nova.servers.list()\n File \"/usr/lib/python2.7/site-packages/novaclient/v2/servers.py\", line 854, in list\n \"servers\")\n File \"/usr/lib/python2.7/site-packages/novaclient/base.py\", line 257, in _list\n resp, body = self.api.client.get(url)\n File \"/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py\", line 304, in get\n return self.request(url, 'GET', **kwargs)\n File \"/usr/lib/python2.7/site-packages/novaclient/client.py\", line 77, in request\n **kwargs)\n File \"/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py\", line 463, in request\n resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py\", line 189, in request\n return self.session.request(url, method, **kwargs)\n File \"/usr/lib/python2.7/site-packages/keystoneauth1/session.py\", line 573, in request\n auth_headers = self.get_auth_headers(auth)\n File \"/usr/lib/python2.7/site-packages/keystoneauth1/session.py\", line 900, in get_auth_headers\n return auth.get_headers(self, **kwargs)\n File \"/usr/lib/python2.7/site-packages/keystoneauth1/plugin.py\", line 95, in get_headers\n token = self.get_token(session)\n File \"/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py\", line 88, in get_token\n return self.get_access(session).auth_token\n File \"/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py\", line 134, in get_access\n self.auth_ref = self.get_auth_ref(session)\n File \"/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py\", line 201, in get_auth_ref\n return self._plugin.get_auth_ref(session, **kwargs)\n File \"/usr/lib/python2.7/site-packages/keystoneauth1/identity/v2.py\", line 63, in get_auth_ref\n authenticated=False, log=False)\n File \"/usr/lib/python2.7/site-packages/keystoneauth1/session.py\", line 848, in post\n return self.request(url, 'POST', **kwargs)\n File \"/usr/lib/python2.7/site-packages/keystoneauth1/session.py\", line 737, in request\n raise exceptions.from_response(resp, method, url)\nkeystoneauth1.exceptions.http.NotFound: (https://192.168.24.2:13000/v2.0/tokens): The resource could not be found. (HTTP 404) (Request-ID: req-fd5dc9ab-718f-4931-884e-f31148c7e1e5)", "stderr_lines": ["Traceback (most recent call last):", " File \"/home/jenkins/get-overcloud-nodes.py\", line 44, in <module>", " for server in nova.servers.list()", " File \"/usr/lib/python2.7/site-packages/novaclient/v2/servers.py\", line 854, in list", " \"servers\")", " File \"/usr/lib/python2.7/site-packages/novaclient/base.py\", line 257, in _list", " resp, body = self.api.client.get(url)", " File \"/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py\", line 304, in get", " return self.request(url, 'GET', **kwargs)", " File \"/usr/lib/python2.7/site-packages/novaclient/client.py\", line 77, in request", " **kwargs)", " File \"/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py\", line 463, in request", " resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)", " File \"/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py\", line 189, in request", " return self.session.request(url, method, **kwargs)", " File \"/usr/lib/python2.7/site-packages/keystoneauth1/session.py\", line 573, in request", " auth_headers = self.get_auth_headers(auth)", " File \"/usr/lib/python2.7/site-packages/keystoneauth1/session.py\", line 900, in get_auth_headers", " return auth.get_headers(self, **kwargs)", " File \"/usr/lib/python2.7/site-packages/keystoneauth1/plugin.py\", line 95, in get_headers", " token = self.get_token(session)", " File \"/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py\", line 88, in get_token", " return self.get_access(session).auth_token", " File \"/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py\", line 134, in get_access", " self.auth_ref = self.get_auth_ref(session)", " File \"/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py\", line 201, in get_auth_ref", " return self._plugin.get_auth_ref(session, **kwargs)", " File \"/usr/lib/python2.7/site-packages/keystoneauth1/identity/v2.py\", line 63, in get_auth_ref", " authenticated=False, log=False)", " File \"/usr/lib/python2.7/site-packages/keystoneauth1/session.py\", line 848, in post", " return self.request(url, 'POST', **kwargs)", " File \"/usr/lib/python2.7/site-packages/keystoneauth1/session.py\", line 737, in request", " raise exceptions.from_response(resp, method, url)", "keystoneauth1.exceptions.http.NotFound: (https://192.168.24.2:13000/v2.0/tokens): The resource could not be found. (HTTP 404) (Request-ID: req-fd5dc9ab-718f-4931-884e-f31148c7e1e5)"], "stdout": "", "stdout_lines": []}

I check the stackrc and seems to be correct, perhaps the keystone v2 service is no longer supported. I believe the TASK [tripleo-inventory : fetch overcloud node names and IPs] could use the os_server_facts ansible instead of a python code, or the get-overcloud-nodes.py should be updated to use v3 session

Arx Cruz (arxcruz)
Changed in tripleo:
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-quickstart (master)

Fix proposed to branch: master
Review: https://review.openstack.org/572709

Changed in tripleo:
assignee: nobody → Arx Cruz (arxcruz)
status: Triaged → In Progress
Revision history for this message
Arx Cruz (arxcruz) wrote :

Same behavior in the following jobs:

periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-queens
periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens
periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens

Revision history for this message
Arx Cruz (arxcruz) wrote :

pike, ocata and master jobs are not having this problem because the fetch overcloud node names and OPs are being skipped on those jobs. Wondering why only queens are not skipping.

Arx Cruz (arxcruz)
Changed in tripleo:
importance: High → Critical
tags: added: promotion-blocker
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-quickstart (master)

Reviewed: https://review.openstack.org/572709
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart/commit/?id=52edd3d3b9acda58f92660531d2c6d776d0d0e2c
Submitter: Zuul
Branch: master

commit 52edd3d3b9acda58f92660531d2c6d776d0d0e2c
Author: Arx Cruz <email address hidden>
Date: Wed Jun 6 11:19:37 2018 +0200

    Fix OS_AUTH_URL

    Stackrc sometimes doesn't bring the /v3 at the end, which is the right
    endpoint for make requests in nova client

    Change-Id: I1b88eef1aab082b9bab06871069c944e9a8201f0
    Closes-Bug: 1775365

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-quickstart 2.1.1

This issue was fixed in the openstack/tripleo-quickstart 2.1.1 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.