Keystone v2.0 APIs have been removed, TripleO config is incomplete

[wes hayutin]

https://logs.rdoproject.org/openstack-periodic-4hr/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset005-master/9155f0e/undercloud/var/log/messages.txt.gz#_Oct__4_16_36_21

Oct 4 16:36:21 upstream-centos-7-2-node-rdo-cloud-tripleo-13630 ironic-inspector: 2017-10-04 16:36:21.320 7163 ERROR ironic_inspector.wsgi_service [-] Periodic clean up of node cache failed: Unauthorized: Unauthorized (HTTP 401)
Oct 4 16:36:21 upstream-centos-7-2-node-rdo-cloud-tripleo-13630 ironic-inspector: 2017-10-04 16:36:21.320 7163 ERROR ironic_inspector.wsgi_service Traceback (most recent call last):
Oct 4 16:36:21 upstream-centos-7-2-node-rdo-cloud-tripleo-13630 ironic-inspector: 2017-10-04 16:36:21.320 7163 ERROR ironic_inspector.wsgi_service File "/usr/lib/python2.7/site-packages/ironic_inspector/wsgi_service.py", line 201, in periodic_clean_up
Oct 4 16:36:21 upstream-centos-7-2-node-rdo-cloud-tripleo-13630 ironic-inspector: 2017-10-04 16:36:21.320 7163 ERROR ironic_inspector.wsgi_service sync_with_ironic()
Oct 4 16:36:21 upstream-centos-7-2-node-rdo-cloud-tripleo-13630 ironic-inspector: 2017-10-04 16:36:21.320 7163 ERROR ironic_inspector.wsgi_service File "/usr/lib/python2.7/site-packages/ironic_inspector/wsgi_service.py", line 209, in sync_with_ironic
Oct 4 16:36:21 upstream-centos-7-2-node-rdo-cloud-tripleo-13630 ironic-inspector: 2017-10-04 16:36:21.320 7163 ERROR ironic_inspector.wsgi_service ironic_nodes = ironic.node.list(limit=0)
Oct 4 16:36:21 upstream-centos-7-2-node-rdo-cloud-tripleo-13630 ironic-inspector: 2017-10-04 16:36:21.320 7163 ERROR ironic_inspector.wsgi_service File "/usr/lib/python2.7/site-packages/ironicclient/v1/node.py", line 143, in list
Oct 4 16:36:21 upstream-centos-7-2-node-rdo-cloud-tripleo-13630 ironic-inspector: 2017-10-04 16:36:21.320 7163 ERROR ironic_inspector.wsgi_service limit=limit)
Oct 4 16:36:21 upstream-centos-7-2-node-rdo-cloud-tripleo-13630 ironic-inspector: 2017-10-04 16:36:21.320 7163 ERROR ironic_inspector.wsgi_service File "/usr/lib/python2.7/site-packages/ironicclient/common/base.py", line 149, in _list_pagination
Oct 4 16:36:21 upstream-centos-7-2-node-rdo-cloud-tripleo-13630 ironic-inspector: 2017-10-04 16:36:21.320 7163 ERROR ironic_inspector.wsgi_service resp, body = self.api.json_request('GET', url)
Oct 4 16:36:21 upstream-centos-7-2-node-rdo-cloud-tripleo-13630 ironic-inspector: 2017-10-04 16:36:21.320 7163 ERROR ironic_inspector.wsgi_service File "/usr/lib/python2.7/site-packages/ironicclient/common/http.py", line 558, in json_request
Oct 4 16:36:21 upstream-centos-7-2-node-rdo-cloud-tripleo-13630 ironic-inspector: 2017-10-04 16:36:21.320 7163 ERROR ironic_inspector.wsgi_service resp = self._http_request(url, method, **kwargs)
Oct 4 16:36:21 upstream-centos-7-2-node-rdo-cloud-tripleo-13630 ironic-inspector: 2017-10-04 16:36:21.320 7163 ERROR ironic_inspector.wsgi_service File "/usr/lib/python2.7/site-packages/ironicclient/common/http.py", line 188, in wrapper
Oct 4 16:36:21 upstream-centos-7-2-node-rdo-cloud-tripleo-13630 ironic-inspector: 2017-10-04 16:36:21.320 7163 ERROR ironic_inspector.wsgi_service return func(self, url, method, **kwargs)
Oct 4 16:36:21 upstream-centos-7-2-node-rdo-cloud-tripleo-13630 ironic-inspector: 2017-10-04 16:36:21.320 7163 ERROR ironic_inspector.wsgi_service File "/usr/lib/python2.7/site-packages/ironicclient/common/http.py", line 540, in _http_request
Oct 4 16:36:21 upstream-centos-7-2-node-rdo-cloud-tripleo-13630 ironic-inspector: 2017-10-04 16:36:21.320 7163 ERROR ironic_inspector.wsgi_service error_json.get('debuginfo'), method, url)
Oct 4 16:36:21 upstream-centos-7-2-node-rdo-cloud-tripleo-13630 ironic-inspector: 2017-10-04 16:36:21.320 7163 ERROR ironic_inspector.wsgi_service Unauthorized: Unauthorized (HTTP 401)
Oct 4 16:36:21 upstream-centos-7-2-node-rdo-cloud-tripleo-13630 ironic-inspector: 2017-10-04 16:36:21.320 7163 ERROR ironic_inspector.wsgi_service
Oct 4 16:36:21 upstream-centos-7-2-node-rdo-cloud-tripleo-13630 sshd[8775]: error: Could not load host key: /etc/ssh/ssh_host_dsa_key

I think, but I'm not sure this is the root cause behind the undercloud install error on master.

https://logs.rdoproject.org/openstack-periodic-4hr/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset005-master/9155f0e/undercloud/home/jenkins/undercloud_install.log.txt.gz#_2017-10-04_16_36_27

[Emilien Macchi (emilienm)]

I think it's 404 on keystone endpoints:

https://logs.rdoproject.org/openstack-periodic-4hr/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset005-master/9155f0e/undercloud/var/log/nova/nova-api.log.txt.gz#_2017-10-04_16_36_27_069

Authorization failed for token: NotFound: (http://192.168.24.1:35357/v2.0/tokens): The resource could not be found. (HTTP 404) (Request-ID: req-f326325b-ec6b-49f7-a118-c5500f9ab05c)

and then it's 401 since token couldn't be get.
Maybe a change in keystone?

[Emilien Macchi (emilienm)]

Probably related to https://github.com/openstack/keystone/commit/087b07bfd4f9e212f9d7b36b707babbf945f374f

[Emilien Macchi (emilienm)]

ok, Keystone v2 auth was removed: https://review.openstack.org/#/c/504465/

[Emilien Macchi (emilienm)]

Ok so here are the things that need to happen:

- undercloud: patch instack-undercloud to add all missing user_domain_name and project_domain_name for each module (nova, neutron, etc), to the "Default value.
- overcloud: very that all user_domain_name and project_domain_name in THT have been defaulted to "Default" for all composable services that talk to Keystone.

[Emilien Macchi (emilienm)]

Another thing to do is to patch all puppet-* modules to make require the user_domain_name and project_domain_name parameters so our users don't fall in this bug...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/509646

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/509646
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=003e373b04c3eaf4c5daade2ef2a3c3324b24abd
Submitter: Jenkins
Branch: master

commit 003e373b04c3eaf4c5daade2ef2a3c3324b24abd
Author: Emilien Macchi <email address hidden>
Date: Wed Oct 4 14:33:43 2017 -0700

    Configure all OpenStack services with default domain

    Keystone upstream removed v2.0 APIs so we can't fallback on v2 endpoints
    anymore and we have to provide domains parameters to use v3 API.

    This patch aims to do it.

    Co-Authored-By: Juan Antonio Osorio Robles <email address hidden>
    Change-Id: Id3fcc24614f6bf67079c07a6296ff371f51a1770
    Closes-Bug: #1721366

[wes hayutin (weshayutin)]

still an issue: https://logs.rdoproject.org/openstack-periodic-4hr/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset006-master/81c2fd6/undercloud/var/log/nova/nova-api.log.txt.gz#_2017-10-05_16_39_19_219

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/509841
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=c0f5264f58fcc01cb7efabdd62298666c2874ed8
Submitter: Jenkins
Branch: master

commit c0f5264f58fcc01cb7efabdd62298666c2874ed8
Author: Emilien Macchi <email address hidden>
Date: Thu Oct 5 10:12:49 2017 -0700

    (retry) configure all OpenStack services with default domain

    Keystone upstream removed v2.0 APIs so we can't fallback on v2 endpoints
    anymore and we have to provide domains parameters to use v3 API.

    This patch aims to do it.

    Change-Id: I42c8fa4025be8d059ed902eaefc51dc0c21dc581
    Co-Authored-By: Juan Antonio Osorio Robles <email address hidden>
    Closes-Bug: #1721366

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/510535

[wes hayutin (weshayutin)]

was reopened due to:
undercloud is still failing
https://logs.rdoproject.org/openstack-periodic-4hr/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/724f50b/undercloud/home/jenkins/undercloud_install.log.txt.gz

Reason: (https://192.168.24.2:13000/v2.0/tokens): The resource could not be found. (HTTP 404) (Request-ID: req-80c47f2a-cb1d-4c3f-8b8e-a7fc3dc500a0)

[Emilien Macchi (emilienm)]

https://review.openstack.org/#/c/510535/ should fix it.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/510535
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=2d72ba511782131e1beeccd5048895206b528208
Submitter: Jenkins
Branch: master

commit 2d72ba511782131e1beeccd5048895206b528208
Author: yatin <email address hidden>
Date: Mon Oct 9 16:51:56 2017 +0530

    Use keystone v3 session with novaclient

    Keystone v2.0 is removed so we have to use keystone v3.
    Use already defined keystone v3 session for creating
    novaclient instance.

    Also, remove the version check for novaclient 6 as we now
    having novaclient >9.0.0.

    Change-Id: I4a98e9dad0e2a7601145c9155191c876f920abfd
    Closes-Bug: #1721366

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/instack-undercloud 8.0.0 release.