Horizon gating on dsvm-integration job is broken due to recent changes in devstack/keystone

[Timur Sufiev]

More importantly, Horizon in devstack is broken too due to the inability to get list of projects / switch the current project for a user, see

DEBUG:keystoneauth.session:Request returned failure status: 404
Unable to retrieve project list.
Traceback (most recent call last):
  File "/home/tsufiev/develop/django_openstack_auth/openstack_auth/user.py", line 314, in authorized_tenants
    is_federated=self.is_federated)
  File "/home/tsufiev/develop/django_openstack_auth/openstack_auth/utils.py", line 325, in get_project_list
    projects = client.projects.list(user=kwargs.get('user_id'))
  File "/home/tsufiev/develop/horizon/.venv/local/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
    return wrapped(*args, **kwargs)
  File "/home/tsufiev/develop/horizon/.venv/local/lib/python2.7/site-packages/keystoneclient/v3/projects.py", line 107, in list
    **kwargs)
  File "/home/tsufiev/develop/horizon/.venv/local/lib/python2.7/site-packages/keystoneclient/base.py", line 75, in func
    return f(*args, **new_kwargs)
  File "/home/tsufiev/develop/horizon/.venv/local/lib/python2.7/site-packages/keystoneclient/base.py", line 383, in list
    self.collection_key)
  File "/home/tsufiev/develop/horizon/.venv/local/lib/python2.7/site-packages/keystoneclient/base.py", line 124, in _list
    resp, body = self.client.get(url, **kwargs)
  File "/home/tsufiev/develop/horizon/.venv/local/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 173, in get
    return self.request(url, 'GET', **kwargs)
  File "/home/tsufiev/develop/horizon/.venv/local/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 330, in request
    resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  File "/home/tsufiev/develop/horizon/.venv/local/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 98, in request
    return self.session.request(url, method, **kwargs)
  File "/home/tsufiev/develop/horizon/.venv/local/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
    return wrapped(*args, **kwargs)
  File "/home/tsufiev/develop/horizon/.venv/local/lib/python2.7/site-packages/keystoneauth1/session.py", line 468, in request
    raise exceptions.from_response(resp, method, url)
NotFound: The resource could not be found. (HTTP 404)

[Timur Sufiev (tsufiev-x)]

The Keystone endpoint url Horizon is using is http://<hostip>:5000/v3, then a request is made to http://<hostip>/identity/users/<userid>/projects to return project list which results in Http404

[Morgan Fainberg (mdrnstm)]

When did this start happening? can you provide a logstash query to show?

Thanks!

[Lin Hua Cheng (lin-hua-cheng)]

The issue is related to the issue Matt reported in: http://lists.openstack.org/pipermail/openstack-dev/2016-February/086852.html

[Lin Hua Cheng (lin-hua-cheng)]

weird, I would expect this patch https://github.com/openstack/django_openstack_auth/commit/e008112d0f964b00bd5cb633262365ebc7f9395b would have fixed this issue.

[Lin Hua Cheng (lin-hua-cheng)]

my bad, didn't notice that it is trying to hit "/identity" might be a different issue.

[Lin Hua Cheng (lin-hua-cheng)]

the right url horizon should be hitting should be: http://<hostip>/identity/v3/users/<userid>/

[Rob Cresswell (robcresswell-deactivatedaccount)]

Two potentially relevant devstack commits:
https://review.openstack.org/#/c/193894/
https://review.openstack.org/#/c/312238/

I've not investigated either yet, but at first glance they seem related. It's unclear to me whether one of the patches has missed something, or whether its just exposed a problem with the URL Horizon is configured to hit.

[Rob Cresswell (robcresswell-deactivatedaccount)]

A clean revert of https://review.openstack.org/#/c/193894/ got my env working again (or it coincidentally worked once)

[Samuel de Medeiros Queiroz (samueldmq)]

As pointed out by Lin, the URL is missing a '/v3' in it.

It should be 'http://<hostip>/identity/v3/users/<userid>/projects' rather than 'http://<hostip>/identity/users/<userid>/projects'

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/321640

[Timur Sufiev (tsufiev-x)]

Not a keystone bug actually. The proper fix consists of updating fix_auth_url_version() in openstack_auth.utils and reusing this function in openstack_dashboard.api.keystone.

[Timur Sufiev (tsufiev-x)]

Here is a patch to django_openstack_auth: https://review.openstack.org/#/c/321639

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/321639
Committed: https://git.openstack.org/cgit/openstack/django_openstack_auth/commit/?id=405cb08207cf87145975d534634165caa701c92b
Submitter: Jenkins
Branch: master

commit 405cb08207cf87145975d534634165caa701c92b
Author: Timur Sufiev <email address hidden>
Date: Thu May 26 19:07:36 2016 +0300

    Fix Keystone url version suffix when webpath is present

    There was a false assumption within utils.fix_auth_url_version()
    routine that everything that goes after hostname:port part of Keystone
    auth_url is could be only version suffix. Once '/identity' webpath was
    enabled in Keystone Apache configuration in Devstack by default, the
    falsehood was exposed and broken all integration tests. This is fixed.

    While debugging fix_auth_url_version() I noticed another side-effect
    of the fix: Horizon no longer needs to specify version suffix inside
    OPENSTACK_KEYSTONE_URL setting, the fixed function works perfectly
    without it. This will be mentioned in release notes for the dependent
    Horizon patch.

    Partial-Bug: #1585682
    Needed-By: Icebfc291ec2b06ed84934c75cfd8c9d91cb2a895
    Change-Id: Iea9b8e8378e6c5fb4c60df0073968d8caf7fbc5e

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/321640
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=caa5e910598707d2cfc56faedb6dff97c574e50c
Submitter: Jenkins
Branch: master

commit caa5e910598707d2cfc56faedb6dff97c574e50c
Author: Timur Sufiev <email address hidden>
Date: Thu May 26 19:09:51 2016 +0300

    Fix Keystone version suffix when Keystone webpath is present

    In this patch the URLs that are coming from service catalog are fixed
    to contain the proper version.

    Update requirements.txt in the same commit, because otherwise
    integration tests won't pass for OpenStack Proposal Bot commit which
    should update DOA version.

    Closes-Bug: #1585682
    Depends-On: Iea9b8e8378e6c5fb4c60df0073968d8caf7fbc5e
    Change-Id: Icebfc291ec2b06ed84934c75cfd8c9d91cb2a895

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/323785

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/323787

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/horizon 10.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/323785
Committed: https://git.openstack.org/cgit/openstack/django_openstack_auth/commit/?id=932ca49e769980043996f5872c6513d9fc68bb48
Submitter: Jenkins
Branch: stable/mitaka

commit 932ca49e769980043996f5872c6513d9fc68bb48
Author: Timur Sufiev <email address hidden>
Date: Thu May 26 19:07:36 2016 +0300

    Fix Keystone url version suffix when webpath is present

    There was a false assumption within utils.fix_auth_url_version()
    routine that everything that goes after hostname:port part of Keystone
    auth_url is could be only version suffix. Once '/identity' webpath was
    enabled in Keystone Apache configuration in Devstack by default, the
    falsehood was exposed and broken all integration tests. This is fixed.

    While debugging fix_auth_url_version() I noticed another side-effect
    of the fix: Horizon no longer needs to specify version suffix inside
    OPENSTACK_KEYSTONE_URL setting, the fixed function works perfectly
    without it. This will be mentioned in release notes for the dependent
    Horizon patch.

    Partial-Bug: #1585682
    Needed-By: Icebfc291ec2b06ed84934c75cfd8c9d91cb2a895
    Change-Id: Iea9b8e8378e6c5fb4c60df0073968d8caf7fbc5e
    (cherry picked from commit 405cb08207cf87145975d534634165caa701c92b)

[OpenStack Infra (hudson-openstack)]

Change abandoned by Timur Sufiev (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/323787