o->p->q undercloud upgrade fails with "Cannot use v2 authentication with domain scope"

[Sofer Athlan-Guyot]

Hi,

deploy ocata, upgrade undercloud to pike then to queens and then:

Hi,

After running the openstack undercloud upgrade from pike to queens. We've encountered errors. Kindly see below logs for the errors.

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 2411, in pre_upgrade
    user_domain_name='Default')
  File "/usr/lib/python2.7/site-packages/os_client_config/__init__.py", line 86, in make_client
    return cloud.get_legacy_client(service_key, constructor)
  File "/usr/lib/python2.7/site-packages/os_client_config/cloud_config.py", line 370, in get_legacy_client
    service_key, min_version=min_version, max_version=max_version)
  File "/usr/lib/python2.7/site-packages/os_client_config/cloud_config.py", line 309, in get_session_endpoint
    endpoint = self._get_highest_endpoint(service_types, kwargs)
  File "/usr/lib/python2.7/site-packages/os_client_config/cloud_config.py", line 267, in _get_highest_endpoint
    return session.get_endpoint(**kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 942, in get_endpoint
    return auth.get_endpoint(self, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 379, in get_endpoint
    allow_version_hack=allow_version_hack, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 270, in get_endpoint_data
    service_catalog = self.get_access(session).service_catalog
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
    self.auth_ref = self.get_auth_ref(session)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 199, in get_auth_ref
    self._plugin = self._do_create_plugin(session)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 188, in _do_create_plugin
    'Cannot use v2 authentication with domain scope')
keystoneauth1.exceptions.discovery.DiscoveryFailure: Cannot use v2 authentication with domain scope
Command 'instack-pre-upgrade-undercloud' returned non-zero exit status 1
clean_up UpgradeUndercloud: Command 'instack-pre-upgrade-undercloud' returned non-zero exit status 1
END return value: 1

Originally found by pvc on #tripleo.

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

So, for the last error, it's as if we're using /v2 path[1] while we're using domain[2]

[1] https://github.com/openstack/keystoneauth/blob/master/keystoneauth1/identity/generic/base.py#L150-L153

[2] https://github.com/openstack/instack-undercloud/blob/stable/queens/instack_undercloud/undercloud.py#L2410..L2411

After updating the OS_AUTH_URL in stackrc to a v3 address, the user could proceed with the undercloud upgrade.

[p.cipriano (ctpeter)]

I updated my stackrc to use auth_url v3 authentication and it is now working.

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

This is the likely change[1] that broke this upgrade path.

[1] https://github.com/openstack/instack-undercloud/commit/96f09de818f9ce099a1db0cb8ccd95f570edc016

[Bogdan Dobrelya (bogdando)]

Note this is won't fix for the master (Stein), nor Rocky (as it has containerized undercloud instead of instack) and should only be fixed as a backports.

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

Hi Bogdan,

I don't understand the new status as it still has to be fixed in pike, so "won't fix" sounds strange to me.

Furthermore even if we don't use it the faulty code is still there in master, so even if we don't use it anymore, it's still there and faulty. I don't really get this neither, but that's a lesser concern to me.

[Bogdan Dobrelya (bogdando)]

It should be fixed only for Pike, thanks!

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

Actually we need queens as well as in pike we don't use stackrc during pike undercloud upgrade so it can be unoticed and then happen only in queens where we do use authentication to check the overcloud state.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/611647

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/611919

[OpenStack Infra (hudson-openstack)]

Change abandoned by Athlan-Guyot sofer (<email address hidden>) on branch: stable/queens
Review: https://review.openstack.org/611647
Reason: legit error

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/611647
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=a1038f30366f95aef7cf0b1df8f695a638fc499e
Submitter: Zuul
Branch: stable/queens

commit a1038f30366f95aef7cf0b1df8f695a638fc499e
Author: Sofer Athlan-Guyot <email address hidden>
Date: Thu Oct 18 12:21:25 2018 +0200

    Update stackrc to v3 auth values when --use-heat is false.

    During pike we have updated the deployment to use v3 authentication in
    stackrc. But we didn't upgrade the existing stackrc.

    We didn't use it in the code so it was not a problem.

    But in queens, when we use '--use-heat' is false, we check the status
    of the overcloud before starting and we load auth url from stackrc and
    force domain in the api request. So we have a v2 url and a v3
    request. That fails.

    Now, we make sure that the stackrc is in v3 version before starting
    the undercloud upgrade in queens when we use --use-heat set to false.

    Change-Id: I3f008b9466408fe797ca50ebf96651a72fc9a544
    Closes-Bug: #1798553

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/instack-undercloud 8.4.7 release.