OAuth Identity token gives Forbidden

[Bogdan]

I have enabled OAuth1 in Keystone Kilo, then followed the flow described here:
https://specs.openstack.org/openstack/keystone-specs/api/v3/identity-api-v3-os-oauth1-ext.html#delegated-authentication-flow

Created a consumer, created a request token, authorized the request token, exchanged it for an access token and finally obtained Identity token out of the access token, which looks like:

HTTP/1.1 201 Created
Date: Thu, 04 Feb 2016 00:20:13 GMT
Server: Apache/2.4.10 (Linux/SUSE)
Content-Length: 7982
X-Subject-Token: 5bae545dc72d499bb3ec2792c9e53cbd
Vary: X-Auth-Token
x-openstack-request-id: req-241f91a2-8bc5-44a0-8676-8f521e074475
Content-Type: application/json

{"token": {"methods": ["oauth1"], "roles": [{"id": "9fe2ff9ee4384b1894a90878d3e92bab", "name": "_member_"}], "expires_at": "2016-02-04T01:20:13.114596Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": ..........skipped catalog, etc..... "OS-OAUTH1": {"access_token_id": "f718a55aeae24fa1930b726cbd41b378", "consumer_id": "979f33d9d2c54fd4ae9d5ed3c2c8f61b"}}}

Then when I try to use the token for example to list servers:

openstack --os-token 5bae545dc72d499bb3ec2792c9e53cbd --os-auth-url https://host:5000/v3 --os-identity-api-version 3 --os-cacert /etc/pki/trust/anchors/ca.pem --os-project-name Project1 server list

I get a surprising error:

Forbidden: You are not authorized to perform the requested action. (Disable debug mode to suppress these details.) (HTTP 403) (Request-ID: req-34f9098e-7f5d-45e6-95b6-6f4cac87159e)

After some debugging I found out that my call gets rejected at:

def token_authenticate(context, auth_payload, user_context, token_ref):
    try:

        # Do not allow tokens used for delegation to
        # create another token, or perform any changes of
        # state in Keystone. To do so is to invite elevation of
        # privilege attacks

        if token_ref.oauth_scoped or token_ref.trust_scoped:
            raise exception.Forbidden()

What am I missing here? My token definitely is oauth_scoped and how am I supposed to use this Identity token?

[Steve Martinelli (stevemar)]

@bogdan, are you able to repeat the problem, but add --help to the openstackclient command? I'm wondering if openstackclient is attempting to re-authenticate instead of just using the token.

[Steve Martinelli (stevemar)]

also, any steps you did to create the request and access tokens would be helpful to have documented as well

[Bogdan (bogdan-vatkov)]

Hi Steve,

I am not sure what --help would give you (at my side it just prints the help without attempting execution of any of the commands), did you mean --debug?

Here is what --debug provides:

DEBUG: cliff.commandmanager found command 'hypervisor_stats_show'
DEBUG: cliff.commandmanager found command 'security_group_create'
DEBUG: cliff.commandmanager found command 'security_group_rule_list'
DEBUG: cliff.commandmanager found command 'keypair_list'
DEBUG: cliff.commandmanager found command 'keypair_delete'
DEBUG: cliff.commandmanager found command 'host_list'
DEBUG: cliff.commandmanager found command 'keypair_create'
DEBUG: cliff.commandmanager found command 'server_pause'
DEBUG: cliff.commandmanager found command 'server_reboot'
DEBUG: cliff.commandmanager found command 'server_migrate'
DEBUG: cliff.commandmanager found command 'server_set'
DEBUG: cliff.commandmanager found command 'host_show'
DEBUG: cliff.commandmanager found command 'server_unrescue'
DEBUG: cliff.commandmanager found command 'usage_list'
DEBUG: cliff.commandmanager found command 'server_add_volume'
DEBUG: cliff.commandmanager found command 'server_unlock'
DEBUG: cliff.commandmanager found command 'security_group_set'
DEBUG: cliff.commandmanager found command 'compute_agent_create'
DEBUG: cliff.commandmanager found command 'server_rescue'
DEBUG: cliff.commandmanager found command 'server_add_security_group'
DEBUG: cliff.commandmanager found command 'console_log_show'
DEBUG: cliff.commandmanager found command 'compute_agent_delete'
DEBUG: cliff.commandmanager found command 'server_ssh'
DEBUG: cliff.commandmanager found command 'server_lock'
DEBUG: cliff.commandmanager found command 'server_unset'
DEBUG: cliff.commandmanager found command 'server_show'
DEBUG: cliff.commandmanager found command 'server_suspend'
DEBUG: cliff.commandmanager found command 'keypair_show'
DEBUG: cliff.commandmanager found command 'server_image_create'
DEBUG: cliff.commandmanager found command 'flavor_list'
DEBUG: cliff.commandmanager found command 'server_remove_volume'
DEBUG: cliff.commandmanager found command 'security_group_delete'
DEBUG: cliff.commandmanager found command 'aggregate_add_host'
DEBUG: cliff.commandmanager found command 'aggregate_remove_host'
DEBUG: cliff.commandmanager found command 'server_remove_security_group'
DEBUG: cliff.commandmanager found command 'ip_floating_remove'
DEBUG: cliff.commandmanager found command 'aggregate_create'
DEBUG: cliff.commandmanager found command 'hypervisor_show'
DEBUG: cliff.commandmanager found command 'ip_floating_list'
DEBUG: cliff.commandmanager found command 'aggregate_delete'
DEBUG: cliff.commandmanager found command 'usage_show'
DEBUG: cliff.commandmanager found command 'security_group_rule_create'
DEBUG: cliff.commandmanager found command 'compute_agent_set'
DEBUG: cliff.commandmanager found command 'server_rebuild'
DEBUG: cliff.commandmanager found command 'flavor_delete'
DEBUG: cliff.commandmanager found command 'server_delete'
DEBUG: cliff.commandmanager found command 'server_resume'
DEBUG: cliff.commandmanager found command 'availability_zone_list'
DEBUG: cliff.commandmanager found command 'hypervisor_list'
DEBUG: cliff.commandmanage...

[Bogdan (bogdan-vatkov)]

One more update. Since I wanted to debug the process I changed the line 172 in

/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v3/base.py

which was
        _logger.debug('Making authentication request to %s', token_url)
so I wanted to adjust it to:
        _logger.debug('Making authentication request to %s with body= %s', token_url, str(body))
but mistyped and wrote:
        _logger.debug('Making authentication request to %s with body= $s', token_url, str(body))
that is why I had this in the error log:

Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 859, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 732, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 471, in format
    record.message = record.getMessage()
  File "/usr/lib64/python2.7/logging/__init__.py", line 335, in getMessage
    msg = msg % self.args
TypeError: not all arguments converted during string formatting
Logged from file base.py, line 172

but after correcting it I have some more info in the log now - instead of the trace above I now have the body logged:

DEBUG: keystoneclient.auth.identity.v3.base Making authentication request to https://host:5000/v3/auth/tokens with body= {'auth': {'scope': {'project': {'domain': {'id': 'default'}, 'name': 'Project1'}}, 'identity': {'token': {'id': '935259242f3c46c6853863394b9882f6'}, 'methods': ['token']}}}

where 935259242f3c46c6853863394b9882f6 is the scoped token.

Sorry for the inconvenience.

[Morgan Fainberg (mdrnstm)]

So, this might be by design. The OAuth1 token would need to be pre-scoped to the project you're working on and you wouldn't use --os-project-name in the CLI. This is because that is effectively rescoping the token (which is not allowed).

If the OAuth token is already scoped to the intended project and --os-project-name is left off the CLI argument does this work as intended (I'm sorry it's been a while since i've looked at this so I need to ask some questions to be sure I am interpreting the bug correctly).

[Bogdan (bogdan-vatkov)]

Hi Morgan,

I do have the project scoping in my token {"token": {"methods": ["oauth1"], "roles": [{"id": "9fe2ff9ee4384b1894a90878d3e92bab", "name": "_member_"}], "expires_at": "2016-02-19T11:50:48.085682Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "f38f21172dcf4dc59660490da8b091f1", "name": .....

And now I tried the CLI without the --os-project-name but it gives me the very same error (Forbidden) and it is caused by the very same lines of code:

        # Do not allow tokens used for delegation to
        # create another token, or perform any changes of
        # state in Keystone. To do so is to invite elevation of
        # privilege attacks

        if token_ref.oauth_scoped or token_ref.trust_scoped:
            raise exception.Forbidden()

What else could be the reason for this behavior?

Thanks in advance!

Best regards,
Bogdan

[Steve Martinelli (stevemar)]

Hey Bogdan, yes, I meant --debug earlier!

Can you try using the generated token in a curl call to list servers?

curl -H "X-Auth-Token: $oauth_scoped_token" -X GET http://nova_endpoint/v2.1/​{tenant_id}​/servers | python -m json.tool

I am wondering why the request is going back to that portion of keystone which is reserved for re-scoping...

[Bogdan (bogdan-vatkov)]

Just wondering if there are any success stories with OAuth authentication documented.
Looking at the code I can't figure out how could it work at all. Could it be that I am missing something?

I am about to re-test the setup with Liberty soon. Will come back with the findings.

[Bogdan (bogdan-vatkov)]

I posted my comment before I see yours Steve, let me test it now...

[Bogdan (bogdan-vatkov)]

Yes, curl request to nova with the oauth scoped token works just fine! What does that mean now? How is the flow that goes to the keystone piece you mentioned happening? Could it be some api-paste.ini misconfiguration?

curl -H "X-Auth-Token: 81fe2525836549039d3ecd25fa5167cc" -X GET http://localhost:8774/v2.1/​f38f21172dcf4dc59660490da8b091f1/servers | python -m json.tool
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 690 100 690 0 0 648 0 0:00:01 0:00:01 --:--:-- 649
{
    "servers": [
        {
            "id": "eaec87f9-6e7c-46c8-8936-39f882843b52",
            "links": [
                {
                    "href": "http://localhost:8774/v2.1/f38f21172dcf4dc59660490da8b091f1/servers/eaec87f9-6e7c-46c8-8936-39f882843b52",
                    "rel": "self"
                },
                {
                    "href": "http://localhost:8774/f38f21172dcf4dc59660490da8b091f1/servers/eaec87f9-6e7c-46c8-8936-39f882843b52",
                    "rel": "bookmark"
                }
            ],
            "name": "pp"
        },
        {
            "id": "571516cb-378c-4eaf-955c-94286c3ae4bc",
            "links": [
                {
                    "href": "http://localhost:8774/v2.1/f38f21172dcf4dc59660490da8b091f1/servers/571516cb-378c-4eaf-955c-94286c3ae4bc",
                    "rel": "self"
                },
                {
                    "href": "http://localhost:8774/f38f21172dcf4dc59660490da8b091f1/servers/571516cb-378c-4eaf-955c-94286c3ae4bc",
                    "rel": "bookmark"
                }
            ],
            "name": "vm1"
        }
    ]
}

[Bogdan (bogdan-vatkov)]

Browsing through the code I do not see much conditions where the call could execute in a different way - the Openstack client always does "POST" to the /v3/auth/tokens API

        resp = session.post(token_url, json=body, headers=headers,
                            authenticated=False, log=False, **rkwargs)

(in /usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v3/base.py)

..., so I do not see how could we expect for the execution not go through the re-scoping code.

[Dolph Mathews (dolph)]

This seems like a missing use case in openstackclient? If you give it an existing token, why is it trying to rescope it? What is it trying to rescope the token to?

[Bogdan (bogdan-vatkov)]

Hi Dolph,

Looking at the code it seems like a missing use case, but this is not what an average OpenStack user would expect - the expectation is that once an auth token is obtained (no matter if using OAuth auth flow or other) it should be possible to use it against the OpenStack APIs as well as the openstack CLI.

If this is intended behavior that one cannot use OAuth tokens with the CLI then it should at least be documented.

Just wondering what the Incomplete status in this case means. I read the description - "Cannot be verified, the reporter needs to give more info."

I followed the official doc on how to obtain an auth token via the OAuth mechanism - did that successfully.
If I try to use the token against the openstackclient I get the failure, if I use it against the REST API directly then it works.

Isn't this explanation enough to get the bug verified? What other information do you need?
I would be glad to help, just tell me how.

Best regards,
Bogdan

[Morgan Fainberg (mdrnstm)]

Marking as incomplete for OSC, please re-visit if it is still an issue (many things have changed across the board) and invalid for keystone.

[Morgan Fainberg (mdrnstm)]

Ah, i can't mark as anything in OSC, someone else may need to circle up on this.