Downstream RDO Phase2 promotion jobs are failing with the following error:
01:53:41 TASK [tripleo-inventory : fetch overcloud node names and IPs] ******************
01:53:41 task path: /home/rhos-ci/jenkins/workspace/oooq-master-rdo_trunk-bmu-haa16-lab-float_nic_with_vlans/tripleo-quickstart/roles/tripleo-inventory/tasks/main.yml:19
01:53:41 Thursday 09 November 2017 20:53:41 -0500 (0:00:02.739) 2:00:11.529 *****
01:53:41 Using module file /home/rhos-ci/jenkins/workspace/oooq-master-rdo_trunk-bmu-haa16-lab-float_nic_with_vlans/lib/python2.7/site-packages/ansible/modules/core/commands/command.py
01:53:41 <undercloud> ESTABLISH SSH CONNECTION FOR USER: stack
01:53:41 <undercloud> SSH: EXEC ssh -vvv -F /home/rhos-ci/jenkins/workspace/oooq-master-rdo_trunk-bmu-haa16-lab-float_nic_with_vlans/ssh.config.ansible -o StrictHostKeyChecking=no -o 'IdentityFile="/home/rhos-ci/jenkins/workspace/oooq-master-rdo_trunk-bmu-haa16-lab-float_nic_with_vlans/id_rsa_undercloud"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=stack -o ConnectTimeout=10 -F /home/rhos-ci/jenkins/workspace/oooq-master-rdo_trunk-bmu-haa16-lab-float_nic_with_vlans/ssh.config.ansible undercloud '/bin/sh -c '"'"'/usr/bin/python && sleep 0'"'"''
01:53:43 fatal: [undercloud]: FAILED! => {
01:53:43 "changed": true,
01:53:43 "cmd": "source /home/stack/stackrc; python /home/stack/get-overcloud-nodes.py",
01:53:43 "delta": "0:00:00.638107",
01:53:43 "end": "2017-11-09 20:53:42.924405",
01:53:43 "failed": true,
01:53:43 "invocation": {
01:53:43 "module_args": {
01:53:43 "_raw_params": "source /home/stack/stackrc; python /home/stack/get-overcloud-nodes.py",
01:53:43 "_uses_shell": true,
01:53:43 "chdir": null,
01:53:43 "creates": null,
01:53:43 "executable": null,
01:53:43 "removes": null,
01:53:43 "warn": true
01:53:43 },
01:53:43 "module_name": "command"
01:53:43 },
01:53:43 "rc": 1,
01:53:43 "start": "2017-11-09 20:53:42.286298",
01:53:43 "stderr": "Traceback (most recent call last):\n File \"/home/stack/get-overcloud-nodes.py\", line 39, in <module>\n for server in nova.servers.list()\n File \"/usr/lib/python2.7/site-packages/novaclient/v2/servers.py\", line 884, in list\n \"servers\")\n File \"/usr/lib/python2.7/site-packages/novaclient/base.py\", line 254, in _list\n resp, body = self.api.client.get(url)\n File \"/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py\", line 304, in get\n return self.request(url, 'GET', **kwargs)\n File \"/usr/lib/python2.7/site-packages/novaclient/client.py\", line 77, in request\n **kwargs)\n File \"/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py\", line 463, in request\n resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py\", line 189, in request\n return self.session.request(url, method, **kwargs)\n File \"/usr/lib/python2.7/site-packages/keystoneauth1/session.py\", line 573, in request\n auth_headers = self.get_auth_headers(auth)\n File \"/usr/lib/python2.7/site-packages/keystoneauth1/session.py\", line 900, in get_auth_headers\n return auth.get_headers(self, **kwargs)\n File \"/usr/lib/python2.7/site-packages/keystoneauth1/plugin.py\", line 90, in get_headers\n token = self.get_token(session)\n File \"/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py\", line 89, in get_token\n return self.get_access(session).auth_token\n File \"/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py\", line 135, in get_access\n self.auth_ref = self.get_auth_ref(session)\n File \"/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py\", line 198, in get_auth_ref\n return self._plugin.get_auth_ref(session, **kwargs)\n File \"/usr/lib/python2.7/site-packages/keystoneauth1/identity/v2.py\", line 63, in get_auth_ref\n authenticated=False, log=False)\n File \"/usr/lib/python2.7/site-packages/keystoneauth1/session.py\", line 848, in post\n return self.request(url, 'POST', **kwargs)\n File \"/usr/lib/python2.7/site-packages/keystoneauth1/session.py\", line 737, in request\n raise exceptions.from_response(resp, method, url)\nkeystoneauth1.exceptions.http.NotFound: (http://192.168.24.1:5000/v2.0/tokens): The resource could not be found. (HTTP 404) (Request-ID: req-2526267b-5441-4252-ae45-0f48802866f3)",
01:53:43 "stdout": "",
01:53:43 "stdout_lines": [],
01:53:43 "warnings": []
01:53:43 }
The issue seems to be that the script is trying to use the keystone v2 endpoint while connecting to the undercloud to get the overcloud node info.
See:
https://github.com/openstack/tripleo-quickstart/blob/master/roles/tripleo-inventory/templates/get-overcloud-nodes.py.j2
https://github.com/openstack/tripleo-quickstart/blob/cf08e93b29c3803cd0e552b5559c90171a161a70/roles/tripleo-inventory/tasks/inventory.yml#L15
This was fixed by this patch [1] that adds domain information on the nova connection, our problem was that we were not up-to-date with the upstream oooq repo.
[1] https:/