Nova api 'Authorization failed for token' with federated scoped token

[William Lynch]

OpenStack Release: Juno

I am investigated k2k and I'm seeing the following behavior

I have setup a keystone 2 keystone environment.
I get a unscoped federated token
I then get a project scoped token from the unscoped.
I attempt to something simple by listing the flavors

-- curl -i -X GET -H "X-Auth-Token:eb2966a9b55e4836907b956b79187341" http://keystone.service.provider:8774/v2/031a04fd26da4d74b9d2375de2d80be0/flavors

I see this in the nova api.log:
-------------------------------

2015-02-06 10:20:32.787 3970 WARNING keystonemiddleware.auth_token [-] Authorization failed for token
2015-02-06 10:20:32.788 3970 INFO nova.osapi_compute.wsgi.server [-] 9.10.111.91 "GET /v2/031a04fd26da4d74b9d2375de2d80be0/flavors HTTP/1.1" status: 401 len: 261 time: 0.0038319

I see this in the keystone.log:
-------------------------------

2015-02-06 10:55:00.753 5910 DEBUG keystone.common.controller [-] RBAC: Authorization granted inner /usr/lib/python2.6/site-packages/keystone/common/controller.py:155
2015-02-06 10:55:00.769 5910 ERROR keystone.common.wsgi [-] 'domain'
2015-02-06 10:55:00.769 5910 TRACE keystone.common.wsgi Traceback (most recent call last):
2015-02-06 10:55:00.769 5910 TRACE keystone.common.wsgi File "/usr/lib/python2.6/site-packages/keystone/common/wsgi.py", line 223, in __call__
2015-02-06 10:55:00.769 5910 TRACE keystone.common.wsgi result = method(context, **params)
2015-02-06 10:55:00.769 5910 TRACE keystone.common.wsgi File "/usr/lib/python2.6/site-packages/keystone/common/controller.py", line 156, in inner
2015-02-06 10:55:00.769 5910 TRACE keystone.common.wsgi return f(self, context, *args, **kwargs)
2015-02-06 10:55:00.769 5910 TRACE keystone.common.wsgi File "/usr/lib/python2.6/site-packages/keystone/token/controllers.py", line 445, in validate_token
2015-02-06 10:55:00.769 5910 TRACE keystone.common.wsgi return self.token_provider_api.validate_v2_token(token_id, belongs_to)
2015-02-06 10:55:00.769 5910 TRACE keystone.common.wsgi File "/usr/lib/python2.6/site-packages/keystone/token/provider.py", line 246, in validate_v2_token
2015-02-06 10:55:00.769 5910 TRACE keystone.common.wsgi token = self._validate_v2_token(token_ref)
2015-02-06 10:55:00.769 5910 TRACE keystone.common.wsgi File "/usr/lib/python2.6/site-packages/dogpile/cache/region.py", line 1008, in decorate
2015-02-06 10:55:00.769 5910 TRACE keystone.common.wsgi should_cache_fn)
2015-02-06 10:55:00.769 5910 TRACE keystone.common.wsgi File "/usr/lib/python2.6/site-packages/dogpile/cache/region.py", line 637, in get_or_create
2015-02-06 10:55:00.769 5910 TRACE keystone.common.wsgi async_creator) as value:
2015-02-06 10:55:00.769 5910 TRACE keystone.common.wsgi File "/usr/lib/python2.6/site-packages/dogpile/core/dogpile.py", line 158, in __enter__
2015-02-06 10:55:00.769 5910 TRACE keystone.common.wsgi return self._enter()
2015-02-06 10:55:00.769 5910 TRACE keystone.common.wsgi File "/usr/lib/python2.6/site-packages/dogpile/core/dogpile.py", line 98, in _enter
2015-02-06 10:55:00.769 5910 TRACE keystone.common.wsgi generated = self._enter_create(createdtime)
2015-02-06 10:55:00.769 5910 TRACE keystone.common.wsgi File "/usr/lib/python2.6/site-packages/dogpile/core/dogpile.py", line 149, in _enter_create
2015-02-06 10:55:00.769 5910 TRACE keystone.common.wsgi created = self.creator()
2015-02-06 10:55:00.769 5910 TRACE keystone.common.wsgi File "/usr/lib/python2.6/site-packages/dogpile/cache/region.py", line 609, in gen_value
2015-02-06 10:55:00.769 5910 TRACE keystone.common.wsgi created_value = creator()
2015-02-06 10:55:00.769 5910 TRACE keystone.common.wsgi File "/usr/lib/python2.6/site-packages/dogpile/cache/region.py", line 1004, in creator
2015-02-06 10:55:00.769 5910 TRACE keystone.common.wsgi return fn(*arg, **kw)
2015-02-06 10:55:00.769 5910 TRACE keystone.common.wsgi File "/usr/lib/python2.6/site-packages/keystone/token/provider.py", line 329, in _validate_v2_token
2015-02-06 10:55:00.769 5910 TRACE keystone.common.wsgi return self.driver.validate_v2_token(token_id)
2015-02-06 10:55:00.769 5910 TRACE keystone.common.wsgi File "/usr/lib/python2.6/site-packages/keystone/token/providers/common.py", line 540, in validate_v2_token
2015-02-06 10:55:00.769 5910 TRACE keystone.common.wsgi self._assert_default_domain(token_ref)
2015-02-06 10:55:00.769 5910 TRACE keystone.common.wsgi File "/usr/lib/python2.6/site-packages/keystone/token/providers/common.py", line 502, in _assert_default_domain
2015-02-06 10:55:00.769 5910 TRACE keystone.common.wsgi if (token_ref['token_data']['token']['user']['domain']['id'] !=
2015-02-06 10:55:00.769 5910 TRACE keystone.common.wsgi KeyError: 'domain'
2015-02-06 10:55:00.769 5910 TRACE keystone.common.wsgi

The token body of the scoped token is:
--------------------------------------

{
    "token": {
        "methods": [
            "saml2"
        ],
        "roles": [
            {
                "id": "29c93633be764f5ba0f5c8a35e676192",
                "name": "admin"
            },
            {
                "id": "9cec6650f92b4c7dadf8dd721c63ca86",
                "name": "service"
            }
        ],
        "expires_at": "2015-02-07T16:16:03.637035Z",
        "project": {
            "domain": {
                "id": "default",
                "name": "Default"
            },
            "id": "031a04fd26da4d74b9d2375de2d80be0",
            "name": "admin"
        },
        "catalog": [
            {
                "endpoints": [
                    {
                        "url": "http://keystone.service.provider:35357/v2.0",
                        "interface": "admin",
                        "region": "Region02",
                        "region_id": "Region02",
                        "id": "ab52a310422a42639ffa19ad7dcd02bf"
                    },
                    {
                        "url": "http://keystone.service.provider:5000/v2.0",
                        "interface": "public",
                        "region": "Region02",
                        "region_id": "Region02",
                        "id": "b0bc001ae06543ba94aa717a21fe6ed7"
                    },
                    {
                        "url": "http://keystone.service.provider:5000/v2.0",
                        "interface": "internal",
                        "region": "Region02",
                        "region_id": "Region02",
                        "id": "e84eaf1089234a53902072059f7d8e33"
                    }
                ],
                "type": "identity",
                "id": "19b22a8988d84f5cb1fa4bd591fa9bec",
                "name": "keystone"
            },
            {
                "endpoints": [
                    {
                        "url": "http://keystone.service.provider:9696",
                        "interface": "public",
                        "region": "Region02",
                        "region_id": "Region02",
                        "id": "66986ddfbe5d417882ddbb4588ea0a2a"
                    },
                    {
                        "url": "http://keystone.service.provider:9696",
                        "interface": "internal",
                        "region": "Region02",
                        "region_id": "Region02",
                        "id": "d47a8bfcb8154618b4d57feafb5fd7ce"
                    },
                    {
                        "url": "http://keystone.service.provider:9696",
                        "interface": "admin",
                        "region": "Region02",
                        "region_id": "Region02",
                        "id": "f237d327d52c44a9902bf0a633c8caf5"
                    }
                ],
                "type": "network",
                "id": "5f50b822e83b495eaa66ffa4853ee67a",
                "name": "neutron"
            },
            {
                "endpoints": [
                    {
                        "url": "http://keystone.service.provider:8776/v2/031a04fd26da4d74b9d2375de2d80be0",
                        "interface": "public",
                        "region": "Region02",
                        "region_id": "Region02",
                        "id": "3c273aca09824e8fb81f6ce6929adb52"
                    },
                    {
                        "url": "http://keystone.service.provider:8776/v2/031a04fd26da4d74b9d2375de2d80be0",
                        "interface": "admin",
                        "region": "Region02",
                        "region_id": "Region02",
                        "id": "4aff46491fcb4fda8d54aa45ab95f8ef"
                    },
                    {
                        "url": "http://keystone.service.provider:8776/v2/031a04fd26da4d74b9d2375de2d80be0",
                        "interface": "internal",
                        "region": "Region02",
                        "region_id": "Region02",
                        "id": "878bee15c2344b34b87590dcc5e329c8"
                    }
                ],
                "type": "volumev2",
                "id": "6d9fb0c614374ad997df7ded7d3c95f4",
                "name": "cinderv2"
            },
            {
                "endpoints": [
                    {
                        "url": "http://keystone.service.provider:8000/v1",
                        "interface": "public",
                        "region": "Region02",
                        "region_id": "Region02",
                        "id": "4179c8f8dc7a40d38298806d1d3203e1"
                    },
                    {
                        "url": "http://keystone.service.provider:8000/v1",
                        "interface": "internal",
                        "region": "Region02",
                        "region_id": "Region02",
                        "id": "512ca87f27b64818bba3322ce162dd06"
                    },
                    {
                        "url": "http://keystone.service.provider:8000/v1",
                        "interface": "admin",
                        "region": "Region02",
                        "region_id": "Region02",
                        "id": "e7fbe151a83c41e49f310eec7de26955"
                    }
                ],
                "type": "cloudformation",
                "id": "7684b358a3ef4337a3778586607e378f",
                "name": "heat-cfn"
            },
            {
                "endpoints": [
                    {
                        "url": "http://keystone.service.provider:8774/v2/031a04fd26da4d74b9d2375de2d80be0",
                        "interface": "admin",
                        "region": "Region02",
                        "region_id": "Region02",
                        "id": "3b5b576551af4620919db2702b56fdf1"
                    },
                    {
                        "url": "http://keystone.service.provider:8774/v2/031a04fd26da4d74b9d2375de2d80be0",
                        "interface": "internal",
                        "region": "Region02",
                        "region_id": "Region02",
                        "id": "3de92054cd2a41a8ab4892acea29a1bf"
                    },
                    {
                        "url": "http://keystone.service.provider:8774/v2/031a04fd26da4d74b9d2375de2d80be0",
                        "interface": "public",
                        "region": "Region02",
                        "region_id": "Region02",
                        "id": "c66d0eead768418492874eaafe02fa57"
                    }
                ],
                "type": "compute",
                "id": "a8390a9c621a45ed9069eb032077cb8f",
                "name": "nova"
            },
            {
                "endpoints": [
                    {
                        "url": "http://keystone.service.provider:8777",
                        "interface": "internal",
                        "region": "Region02",
                        "region_id": "Region02",
                        "id": "3642e8b2404f4e31ad7bed7316176a74"
                    },
                    {
                        "url": "http://keystone.service.provider:8777",
                        "interface": "admin",
                        "region": "Region02",
                        "region_id": "Region02",
                        "id": "384d2b4982e84eb4a6d552c3c502943b"
                    },
                    {
                        "url": "http://keystone.service.provider:8777",
                        "interface": "public",
                        "region": "Region02",
                        "region_id": "Region02",
                        "id": "b0694e4208074ed28745db6c9608389e"
                    }
                ],
                "type": "metering",
                "id": "dd0f2f66ca624408bc82c7eb55ef65b1",
                "name": "ceilometer"
            },
            {
                "endpoints": [
                    {
                        "url": "http://keystone.service.provider:8004/v1/031a04fd26da4d74b9d2375de2d80be0",
                        "interface": "internal",
                        "region": "Region02",
                        "region_id": "Region02",
                        "id": "218337fa2dc44b3b9cee2368a399ed56"
                    },
                    {
                        "url": "http://keystone.service.provider:8004/v1/031a04fd26da4d74b9d2375de2d80be0",
                        "interface": "public",
                        "region": "Region02",
                        "region_id": "Region02",
                        "id": "5d25864dde854bfc83839076bc30e774"
                    },
                    {
                        "url": "http://keystone.service.provider:8004/v1/031a04fd26da4d74b9d2375de2d80be0",
                        "interface": "admin",
                        "region": "Region02",
                        "region_id": "Region02",
                        "id": "ab5f112637a34ba2833a377360b1b5a9"
                    }
                ],
                "type": "orchestration",
                "id": "e0b42767b3f247ee8124ab5bbbf232eb",
                "name": "heat"
            },
            {
                "endpoints": [
                    {
                        "url": "http://keystone.service.provider:9292",
                        "interface": "internal",
                        "region": "Region02",
                        "region_id": "Region02",
                        "id": "b5d954ae40b5496bb476cbd7010aabdc"
                    },
                    {
                        "url": "http://keystone.service.provider:9292",
                        "interface": "public",
                        "region": "Region02",
                        "region_id": "Region02",
                        "id": "cca308367f274c5cbe91cd540bb5ee9c"
                    },
                    {
                        "url": "http://keystone.service.provider:9292",
                        "interface": "admin",
                        "region": "Region02",
                        "region_id": "Region02",
                        "id": "e055c8c3d94c4201907a70708a0c177a"
                    }
                ],
                "type": "image",
                "id": "e6a478cfa5e24bcfa3c876b745fad4ed",
                "name": "glance"
            },
            {
                "endpoints": [
                    {
                        "url": "http://keystone.service.provider:8776/v1/031a04fd26da4d74b9d2375de2d80be0",
                        "interface": "internal",
                        "region": "Region02",
                        "region_id": "Region02",
                        "id": "35b10b4ebd26424d98a7b09a7d623783"
                    },
                    {
                        "url": "http://keystone.service.provider:8776/v1/031a04fd26da4d74b9d2375de2d80be0",
                        "interface": "admin",
                        "region": "Region02",
                        "region_id": "Region02",
                        "id": "867b7b6699d041eaad407f629d8c73b3"
                    },
                    {
                        "url": "http://keystone.service.provider:8776/v1/031a04fd26da4d74b9d2375de2d80be0",
                        "interface": "public",
                        "region": "Region02",
                        "region_id": "Region02",
                        "id": "d7eca9c32e594ce8b84afe8784abcf0e"
                    }
                ],
                "type": "volume",
                "id": "f3b990f618424bac91d153bd1b3190d1",
                "name": "cinder"
            }
        ],
        "extras": {},
        "user": {
            "OS-FEDERATION": {
                "identity_provider": {
                    "id": "Wisconsin"
                },
                "protocol": {
                    "id": "saml2"
                }
            },
            "id": "admin",
            "name": "admin"
        },
        "audit_ids": [
            "HY3ENddAQRCkbX68mkI7Uw"
        ],
        "issued_at": "2015-02-06T16:16:03.637076Z"
    }
}

[Steve Martinelli (stevemar)]

the part of the stacktrace that looks suspicious is that it's failing at `validate_v2_token`, i'm wondering if this is caused by nova/keystonemiddleware incorrectly using v2 endpoints?

[Steve Martinelli (stevemar)]

@Bill, do any nova specific functions work (like listing intances)? Not glace functions, like listing flavors... it'll help in narrowing down whats going on

[Marek Denis (marek-denis)]

Hi,

What is the version of Service Provider Keystone? Is it Juno too? Indeed it looks like v2 api is being used and causing the failures. I also saw only v2 identity endpoints in Service Catalog, so maybe it's a matter of api version configuriation?

[William Lynch (wmlynch)]

2 points:

1) I found that the auth_version value in the nova.conf was set to v2.0. I changed the value for the property and things now work in this case. I'm now able to proceed.

2) During a side discussion with Brant, he felt that the way it failed could be improved upon ie) the code should handle this situation more gracefully.

[Marek Denis (marek-denis)]

Hi Willian,

Glad it worked. What improvements are you thinking of? Some warning, because V3 only functionality was being used with V2 API ?

Thanks,

Marek

[Brant Knudson (blk-u)]

The failure shouldn't be "KeyError: 'domain'" -- since tokens might not have a domain, the code should check if 'domain' isn't in the dict and fail with a more appropriate error message... could just be change to .get('domain') rather than ['domain']

[Steve Martinelli (stevemar)]

We can check for an OS-FEDERATION section in the ['user'] block, and raise an error at that point.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/154368

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/154368
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=2d119010e10ee92bb6d181f697ec20c2f76f9aea
Submitter: Jenkins
Branch: master

commit 2d119010e10ee92bb6d181f697ec20c2f76f9aea
Author: Steve Martinelli <email address hidden>
Date: Tue Feb 10 02:51:33 2015 -0500

    Add a check to see if a federation token is being used for v2 auth

    If a user has a federation environment, and doesn't update nova
    to use v3 auth in auth_token, then they will automatically be
    using v2 auth, which causes a fairly useless error to bubble up.
    We should check if the user is using a fedration token in v2 and
    provider a better error message.

    Change-Id: I9c490df8dd38e9fa1f393ebfa1296b64fb0c9527
    Closes-Bug: #1419114