keystone will not auth users if there is a bad endpoint

[Matt Fischer]

I deployed a bad endpoint today and Keystone's failure case was that it was refusing to authenticate users. This was a rather severe failure for a bad swift admin URL. An error level log is fine, but I'd prefer not to impact the rest of my users.

2014-07-23 16:33:39.435 6722 ERROR keystone.catalog.core [-] Malformed endpoint http://foo.com:80/v1/KEY_%{tenant_id)s - incomplete format (are you missing a type notifier ?)

The {tenant_ids) is incorrect obviously.

[Juan Antonio Osorio Robles (juan-osorio-robles)]

Reading the description you gave, it would appear that users aren't able to authenticate at all,
but if that's the case, I couldn't reproduce the error, since even though I created a malformed
endpoint, I was still able to authenticate using another endpoint.

But, if you're trying to authenticate using that malformed endpoint, then I think it's not an issue,
since if the endpoint is wrong it should rightly mark it as such; as is the case.

Can you give more information regarding this issue? Like what version of keystone are you using, and perhaps
how your endpoint list looks like.

[Dolph Mathews (dolph)]

Is this on master or a stable/* release? I added David Stanek because I believe he recently made a change to discard invalid endpoints instead of failing outright. Perhaps that needs to be backported to icehouse?

[Matt Fischer (mfisch)]

It was on icehouse. I've not tried to repo it in devstack.

[David Stanek (dstanek)]

It would make sense that this doesn't work. The fix I made prevented 500 errors when getting a catalog with invalid endpoints. I'll setup an icehouse environment to test this scenario and reports my finding.

[David Stanek (dstanek)]

I was able to reproduce this as I expected. I cherry-picked the fixes I made for the related bug listed below and the malformed endpoint no longer caused Keystone auth to fail.

Related bug: https://bugs.launchpad.net/keystone/+bug/1230279

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/111519

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/111520

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/111521

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/111522

[OpenStack Infra (hudson-openstack)]

Change abandoned by David Stanek (<email address hidden>) on branch: stable/icehouse
Review: https://review.openstack.org/111522
Reason: This was squashed into: https://review.openstack.org/111519

[OpenStack Infra (hudson-openstack)]

Change abandoned by David Stanek (<email address hidden>) on branch: stable/icehouse
Review: https://review.openstack.org/111521
Reason: This was squashed into: https://review.openstack.org/111519

[OpenStack Infra (hudson-openstack)]

Change abandoned by David Stanek (<email address hidden>) on branch: stable/icehouse
Review: https://review.openstack.org/111520
Reason: This was squashed into: https://review.openstack.org/111519

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/111519
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=126440d8da9abfa6e7316c1b44541eb734988de4
Submitter: Jenkins
Branch: stable/icehouse

commit 126440d8da9abfa6e7316c1b44541eb734988de4
Author: David Stanek <email address hidden>
Date: Tue Mar 18 14:05:52 2014 +0000

    Ignore broken endpoints in get_v3_catalog

    Change-Id: Ifd858e9f37155e3806329c4688be494d0132a9c7
    Partial-bug: #1347862
    (cherry picked from commit 7e4aab33e2d38a87e26f9f4e65b4ba2097564fd2)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/111520
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=c959cbdc938fa9a535b1fdce023abc632376fa94
Submitter: Jenkins
Branch: stable/icehouse

commit c959cbdc938fa9a535b1fdce023abc632376fa94
Author: David Stanek <email address hidden>
Date: Wed Mar 19 04:25:18 2014 +0000

    Ignore broken endpoints in get_catalog

    If an endpoint is created with a malformed URL then get_catalog would
    throw a HTTP 500 error. With this change we ignore all malformed
    endpoints.

    Change-Id: Ibe9a8fa49f410f2a76a0df732247bd6813fc734b
    Partial-bug: #1347862
    (cherry picked from commit 8737b43d2cc9cbc1af1ea7a0fbff39f1b1c50747)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/111521
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=f3198d886907d6cbc5081161f0c156e29a2946d9
Submitter: Jenkins
Branch: stable/icehouse

commit f3198d886907d6cbc5081161f0c156e29a2946d9
Author: David Stanek <email address hidden>
Date: Fri Jun 13 18:13:03 2014 +0000

    Updates keystone.catalog.core.format_url tests

    Since the tests were unit tests I moved them into the new directory
    structure. I also added a few tests to increase the coverage to 100%.

    Change-Id: I063f0712b83ddd7773458a5bb3d4f483c20eb2a2
    Partial-bug: #1347862
    (cherry picked from commit 7c1a3421077a1971742e7e9887793ad5f4c9f615)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/111522
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=80502d3ed930049ccab757b870ffa18b70d8551a
Submitter: Jenkins
Branch: stable/icehouse

commit 80502d3ed930049ccab757b870ffa18b70d8551a
Author: David Stanek <email address hidden>
Date: Fri Jun 13 18:28:36 2014 +0000

    Fixes catalog URL formatting to never return None

    The old behavior was to return None if the URL template was not a string
    (or not string-like). This was a mistake because a None should never be
    able to make its way into the catalog.

    This is based off of an IRC discussion that spawned from:
    https://review.openstack.org/#/c/81528/4/keystone/catalog/backends/sql.py

    Change-Id: I12b0362d3869a3ec8dc1a6fa34e934a221deecbc
    Partial-bug: #1347862
    (cherry picked from commit 8599aa5d8393e3217e04bd8fb4bca2b2eaa41a69)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/121711

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ryan Hsu (<email address hidden>) on branch: master
Review: https://review.openstack.org/121711
Reason: Testing