Horizon will not authenticate against keystone v3

[Chris Jones]

Using devstack.
In /opt/stack/horizon/horizon/openstack_dashboard/local/local_settings.py

Setting the following

OPENSTACK_API_VERSIONS = {
    "identity": 3
}

results in an authentication failure in keystone.

A keystone v3 endpoint is available.

What follows are the keystone logs for the failure case:

(eventlet.wsgi.server): 2014-01-09 14:07:04,229 INFO log write (28305) accepted ('127.0.0.1', 41111)

(routes.middleware): 2014-01-09 14:07:04,231 DEBUG middleware __call__ Matched POST /auth/tokens
(routes.middleware): 2014-01-09 14:07:04,231 DEBUG middleware __call__ Route path: '{path_info:.*}', defaults: {'controller': <keystone.contrib.user_crud.core.CrudExtension object at 0x27757d0>}
(routes.middleware): 2014-01-09 14:07:04,232 DEBUG middleware __call__ Match dict: {'controller': <keystone.contrib.user_crud.core.CrudExtension object at 0x27757d0>, 'path_info': '/auth/tokens'}
(routes.middleware): 2014-01-09 14:07:04,232 DEBUG middleware __call__ Matched POST /auth/tokens
(routes.middleware): 2014-01-09 14:07:04,232 DEBUG middleware __call__ Route path: '{path_info:.*}', defaults: {'controller': <keystone.common.wsgi.ComposingRouter object at 0x2775e50>}
(routes.middleware): 2014-01-09 14:07:04,232 DEBUG middleware __call__ Match dict: {'controller': <keystone.common.wsgi.ComposingRouter object at 0x2775e50>, 'path_info': '/auth/tokens'}
(routes.middleware): 2014-01-09 14:07:04,232 DEBUG middleware __call__ No route matched for POST /auth/tokens
(access): 2014-01-09 14:07:04,233 INFO core __call__ 127.0.0.1 - - [09/Jan/2014:22:07:04 +0000] "POST http://127.0.0.1:5000/v2.0/auth/tokens HTTP/1.0" 404 93
(eventlet.wsgi.server): 2014-01-09 14:07:04,233 INFO log write 127.0.0.1 - - [09/Jan/2014 14:07:04] "POST /v2.0/auth/tokens HTTP/1.1" 404 228 0.002791

When using the default (v2.0) keystone (having the above code commented out), authentication succeeds:

What follows are the corresponding partial keystone logs for the success case:

(eventlet.wsgi.server): 2014-01-09 14:08:41,806 INFO log write (28305) accepted ('127.0.0.1', 41112)

(routes.middleware): 2014-01-09 14:08:41,807 DEBUG middleware __call__ Matched POST /tokens
(routes.middleware): 2014-01-09 14:08:41,807 DEBUG middleware __call__ Route path: '{path_info:.*}', defaults: {'controller': <keystone.contrib.user_crud.core.CrudExtension object at 0x27757d0>}
(routes.middleware): 2014-01-09 14:08:41,807 DEBUG middleware __call__ Match dict: {'controller': <keystone.contrib.user_crud.core.CrudExtension object at 0x27757d0>, 'path_info': '/tokens'}
(routes.middleware): 2014-01-09 14:08:41,807 DEBUG middleware __call__ Matched POST /tokens
(routes.middleware): 2014-01-09 14:08:41,807 DEBUG middleware __call__ Route path: '{path_info:.*}', defaults: {'controller': <keystone.common.wsgi.ComposingRouter object at 0x2775e50>}
(routes.middleware): 2014-01-09 14:08:41,808 DEBUG middleware __call__ Match dict: {'controller': <keystone.common.wsgi.ComposingRouter object at 0x2775e50>, 'path_info': '/tokens'}
(routes.middleware): 2014-01-09 14:08:41,808 DEBUG middleware __call__ Matched POST /tokens
(routes.middleware): 2014-01-09 14:08:41,808 DEBUG middleware __call__ Route path: '/tokens', defaults: {'action': u'authenticate', 'controller': <keystone.token.controllers.Auth object at 0x2775f50>}
(routes.middleware): 2014-01-09 14:08:41,808 DEBUG middleware __call__ Match dict: {'action': u'authenticate', 'controller': <keystone.token.controllers.Auth object at 0x2775f50>}
(keystone.common.wsgi): 2014-01-09 14:08:41,808 DEBUG wsgi __call__ arg_dict: {}
(keystone.openstack.common.versionutils): 2014-01-09 14:08:41,809 WARNING log deprecated Deprecated: v2 API is deprecated as of Icehouse in favor of v3 API and may be removed in K.
(dogpile.core.dogpile): 2014-01-09 14:08:41,809 DEBUG dogpile _enter NeedRegenerationException

Using (eventlet.wsgi.server): 2014-01-09 14:08:41,806 INFO log write (28305) accepted ('127.0.0.1', 41112)

(routes.middleware): 2014-01-09 14:08:41,807 DEBUG middleware __call__ Matched POST /tokens
(routes.middleware): 2014-01-09 14:08:41,807 DEBUG middleware __call__ Route path: '{path_info:.*}', defaults: {'controller': <keystone.contrib.user_crud.core.CrudExtension object at 0x27757d0>}
(routes.middleware): 2014-01-09 14:08:41,807 DEBUG middleware __call__ Match dict: {'controller': <keystone.contrib.user_crud.core.CrudExtension object at 0x27757d0>, 'path_info': '/tokens'}
(routes.middleware): 2014-01-09 14:08:41,807 DEBUG middleware __call__ Matched POST /tokens
(routes.middleware): 2014-01-09 14:08:41,807 DEBUG middleware __call__ Route path: '{path_info:.*}', defaults: {'controller': <keystone.common.wsgi.ComposingRouter object at 0x2775e50>}
(routes.middleware): 2014-01-09 14:08:41,808 DEBUG middleware __call__ Match dict: {'controller': <keystone.common.wsgi.ComposingRouter object at 0x2775e50>, 'path_info': '/tokens'}
(routes.middleware): 2014-01-09 14:08:41,808 DEBUG middleware __call__ Matched POST /tokens
(routes.middleware): 2014-01-09 14:08:41,808 DEBUG middleware __call__ Route path: '/tokens', defaults: {'action': u'authenticate', 'controller': <keystone.token.controllers.Auth object at 0x2775f50>}
(routes.middleware): 2014-01-09 14:08:41,808 DEBUG middleware __call__ Match dict: {'action': u'authenticate', 'controller': <keystone.token.controllers.Auth object at 0x2775f50>}
(keystone.common.wsgi): 2014-01-09 14:08:41,808 DEBUG wsgi __call__ arg_dict: {}
(keystone.openstack.common.versionutils): 2014-01-09 14:08:41,809 WARNING log deprecated Deprecated: v2 API is deprecated as of Icehouse in favor of v3 API and may be removed in K.
(dogpile.core.dogpile): 2014-01-09 14:08:41,809 DEBUG dogpile _enter NeedRegenerationException

Fails with this keystone/horizon combination:

Keystone commit hash: f33ec9e5bcd82e0debfa815c7ab551c79854f841
(Jan 7)
Horizon commit hash: 1dd9ec17ed81873057c3423b8632d13d159da64a
(Nov 18)

As well as with a keystone from around the same era:
Keystone commit hash: 2ab2c624353067ba0989720414e5cde2d4792bcc
(Nov 15)

Cheers,
Chris

[David Lyle (david-lyle)]

As Keystone v2 is now deprecated this is a high priority problem.

[Juan Manuel Ollé (juan-m-olle)]

I think

OPENSTACK_KEYSTONE_URL = "http://%s:5000/v3" % OPENSTACK_HOST

must be updates in settings.py

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/66024

[David Lyle (david-lyle)]

At this point, which authentication endpoint we use is not the underlying problem. We will want to change it once we've fixed the larger problem. But this is a recent development and login has worked previously. It is overridden in the code when accessed now.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/68288

[Chris Jones (5-cjones)]

I think Juan was correct here. We should make v3 the default. You need to set both
OPENSTACK_API_VERSIONS and the OPENSTACK_KEYSTONE_URL.

When this is done, there is still some odd behaviour that should be investigated here:
https://bugs.launchpad.net/horizon/+bug/1231357

I believe this is a secondary issue to this one and I will investigate after this change successfully makes it in.
The change is available for review here:
https://review.openstack.org/#/c/68288/

[Julie Pichon (jpichon)]

Clarifying the status since it wasn't obvious to me from the bug comments: reading the review comments on https://review.openstack.org/#/c/68288/ , it looks like the main issue described here was resolved in the django openstack auth project at https://review.openstack.org/#/c/70479/ (now merged). The open review against horizon was part of the fix (updating the sample settings) but has expired.

Probably this bug should have a task open against django_openstack_auth as well?

[David Lyle (david-lyle)]

Bug is fixed in master of django_openstack_auth. I'll release it and close this bug. The fix in django_openstack_auth works even if the settings is mismatched. Commit was https://github.com/openstack/django_openstack_auth/commit/537fd8c7b242d4de5e0f7a30729b59e7bf90a7f1

[Julie Pichon (jpichon)]

Thanks David!

[Akihiro Motoki (amotoki)]

For clarification, I added django-openstack-auth to the affected project. Thanks for the fix.