Projects dropdown fails due to incomplete Keystone endpoint URL

[Johannes Grassler]

Problem Description
===================

The 'Projects' dropdown in our Horizon dashboard (the one on the top
left) is empty since we switched to Openstack Kilo. We investigated the
issue and found the following error message in horizon.log:

  AuthorizationFailure: Authorization Failed: The resource could not be found. (HTTP 404) http://10.0.81.10:5000

[You will find a full stack trace in stacktrace.txt in the attached
tarball.]

Further investigation (see keystone.pcap in the attached tarball for a
packet trace) revealed that horizon is trying to access
http://10.0.81.10:5000/tokens (as opposed to the correct URL,
http://10.0.81.10:5000/v2.0/tokens). We found the problem could be
worked around by appending missing versioning information in backend.py
(see below), but it should be possible to fix this in a cleaner manner.

Environment
===========

We are running Openstack Kilo with the Ubuntu Cloud packages, some of
them modified locally with backported bugfixes. You will find these
packages at https://launchpad.net/~syseleven-platform/+archive/ubuntu/kilo.
In particular, we are running the following Horizon package:

https://launchpad.net/~syseleven-platform/+archive/ubuntu/kilo

Configuration
=============

You will find our full Horizon configuration in local_settings.py in the
attached tarball. Relevant points:

* OPENSTACK_KEYSTONE_URL is http://10.0.81.10:5000/v2.0
* OPENSTACK_API_VERSIONS is configured for Keystone 2.0
* The identity endpoints as reported by Keystone itself do not contain
  versioning information (the way it is supposed to be as of Kilo).

Steps to reproduce
==================

* Run Horizon/Kilo (with the Ubuntu Cloud packages or our modified
  packages; both should exhibit this problem)
* Configure the end points and OPENSTACK_KEYSTONE_URL as described under
  "Configuration"
* Log into the web interface.

This should yield an empty Projects dropdown list and the stacktrace in
stacktrace.txt in /var/log/horizon/horizon.log).

Workaround
==========

I modified /usr/lib/python2.7/dist-packages/openstack_auth/backend.py to
append versioning information to the endpoint URL if it is missing. This
can be used to work around the problem in a pinch, but I do not consider
it a clean fix.

Files
=====

I attached a couple of files to illustrate the problem (you will find
all of these in horizon-projects-dropdown.tar):

backend.py The modified backend.py described under "Workaround"
endpoints.txt A list of identity endpoints as reported by Keystone
keystone.pcap A packet capture of Horizon's interactions with
                    Keystone
local_settings.py Our Horizon configuration
stacktrace.txt The stack trace that appears in
                    /var/log/horizon/horizon.log

[Johannes Grassler (jgr-launchpad)]

We figured out that this line is the problem:

https://github.com/openstack/django_openstack_auth/blob/stable/kilo/openstack_auth/backend.py#L183

It is unchanged from Juno, which means it breaks in the versionless endpoint environment introduced with Kilo. I hacked up a slightly better fix than the workaround up there, but I'm unsure where/how to submit it for review (since it is a change to django_openstack_auth and not to horizon proper).

[Johannes Grassler (jgr-launchpad)]

I've been a bit messy about the change with my patch in it, I'm afraid: the fix requires a patch against django-openstack-auth, but this ticket is a horizon ticket. Apparently this confused review.openstack.org enough to not link the change here.

Anyway, this is the change in question, and it fixes the issue at hand: https://review.openstack.org/#/c/238419/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/238419
Committed: https://git.openstack.org/cgit/openstack/django_openstack_auth/commit/?id=58ce9d7edeac81dbd7e7f77efde33c97e8d1e00c
Submitter: Jenkins
Branch: master

commit 58ce9d7edeac81dbd7e7f77efde33c97e8d1e00c
Author: Johannes Grassler <email address hidden>
Date: Wed Oct 21 17:43:35 2015 +0200

    Add API version to identity endpoint URLs

    This change adds the Keystone API version to the identity endpoint URL
    retrieved from Keystone's endpoint list. This is neccessary in Kilo and
    later, since identity endpoint URLs retrieved from Keystone no longer
    contain the API version path they used to contain until Juno. See
    https://bugs.launchpad.net/horizon/+bug/1508421 for a detailed analysis
    of the problem.

    Change-Id: Ieff5a6cdd1ad352a9731d46785802e8c36adcdd1
    Closes-Bug: 1508421

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/265756

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/265756
Committed: https://git.openstack.org/cgit/openstack/django_openstack_auth/commit/?id=b8f342fb502b5e92ec4261fc558865b1edfc663e
Submitter: Jenkins
Branch: stable/liberty

commit b8f342fb502b5e92ec4261fc558865b1edfc663e
Author: Johannes Grassler <email address hidden>
Date: Wed Oct 21 17:43:35 2015 +0200

    Add API version to identity endpoint URLs

    This change adds the Keystone API version to the identity endpoint URL
    retrieved from Keystone's endpoint list. This is neccessary in Kilo and
    later, since identity endpoint URLs retrieved from Keystone no longer
    contain the API version path they used to contain until Juno. See
    https://bugs.launchpad.net/horizon/+bug/1508421 for a detailed analysis
    of the problem.

    Closes-Bug: 1508421
    (cherry picked from commit 58ce9d7edeac81dbd7e7f77efde33c97e8d1e00c)
    Change-Id: I771555bc3133bc7e140822511ccd12aff8c35fca

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/276930

[Thomas Bechtold (toabctl)]

For the record: The proposed fix from https://git.openstack.org/cgit/openstack/django_openstack_auth/commit/?id=58ce9d7edeac81dbd7e7f77efde33c97e8d1e00c is incomplete. The fixed duplicates the "v3" url if "v3" is already in the url. The result from the keystone.log is "POST /v3/v3/auth/tokens HTTP/1.1" . The review from comment#7 fixes it.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/276930
Committed: https://git.openstack.org/cgit/openstack/django_openstack_auth/commit/?id=e008112d0f964b00bd5cb633262365ebc7f9395b
Submitter: Jenkins
Branch: master

commit e008112d0f964b00bd5cb633262365ebc7f9395b
Author: Thomas Bechtold <email address hidden>
Date: Fri Feb 5 22:52:32 2016 +0100

    Fix "Add API version to identity endpoint URLs"

    Change Ieff5a6cdd1ad352a9731d46785802e8c36adcdd1 introduced an
    uncomplete fix when trying to fix the auth_url.
    Given the case that a auth url already has a version included, an extra
    version was added. This leads to messages in the keystone.log that
    horizon is trying to authenticate with "POST /v3/v3/auth/tokens
    HTTP/1.1".
    Use urlparse correctly and also add a testcase for fix_auth_url_version().

    Change-Id: I80fb310d95e8fdab1212fc5b092a37fd7b26a37a
    Closes-Bug: 1508421

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/280546

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/280546
Committed: https://git.openstack.org/cgit/openstack/django_openstack_auth/commit/?id=3f035859bd6b52e1e25ecdb163ff76c5a8509d68
Submitter: Jenkins
Branch: stable/liberty

commit 3f035859bd6b52e1e25ecdb163ff76c5a8509d68
Author: Thomas Bechtold <email address hidden>
Date: Fri Feb 5 22:52:32 2016 +0100

    Fix "Add API version to identity endpoint URLs"

    Change Ieff5a6cdd1ad352a9731d46785802e8c36adcdd1 introduced an
    uncomplete fix when trying to fix the auth_url.
    Given the case that a auth url already has a version included, an extra
    version was added. This leads to messages in the keystone.log that
    horizon is trying to authenticate with "POST /v3/v3/auth/tokens
    HTTP/1.1".
    Use urlparse correctly and also add a testcase for fix_auth_url_version().

    Change-Id: I80fb310d95e8fdab1212fc5b092a37fd7b26a37a
    (cherry picked from commit e008112d0f964b00bd5cb633262365ebc7f9395b)
    Closes-Bug: 1508421

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/django_openstack_auth 2.2.0 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/289927

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/289937

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matthias Runge (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/289937
Reason: openstack_auth in kilo should be EOLed, following horizon kilo EOL

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matthias Runge (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/289927
Reason: openstack_auth in kilo should be EOLed, following horizon kilo EOL