wrong password set in api-paste.ini, but still pass the auth

[li,chen]

I'm working on Grizzly, and I saw a really strange phenomenon in keystone log.

When I run command "nova list", I get two INFO output:
2013-06-19 15:01:26 INFO [access] 192.168.11.12 - - [19/Jun/2013:07:01:26 +0000] "POST http://keystone:5000/v2.0/tokens HTTP/1.0" 200 5143
2013-06-19 15:01:26 INFO [access] 192.168.11.11 - - [19/Jun/2013:07:01:26 +0000] "GET http://keystone:35357/v2.0/tokens/revoked HTTP/1.0" 200 504

I think this matches my understanding about how auth work, although I have questions about the "revoked".
First, user get a new token, then nova verify the token.

Then, suddenly, the second log disappeared, I can only get:
2013-06-20 16:35:45 INFO [access] 192.168.11.12 - - [20/Jun/2013:08:35:45 +0000] "POST http://keystone:5000/v2.0/tokens HTTP/1.0" 200 5143

This come to me a question, how nova-api verify user's token ?
So, I edited /etc/nova/api-paste.ini, changed admin_password to a wrong number.
Also, I cleaned all tokens in keystone (directly going to database, delete all of them by hand), and restart nova-api.
I suppose this will cause "nova list" failed in auth.
But, I still get my instance list.

I have set "auth_strategy = keystone" in nova.conf.
And before I clean database's token table, I checked the revoke query in keystone log, it contains tokens for both user and nova.

And, I have restarted nova-api, there is a log like:
2013-06-20 17:41:23.505 4605 DEBUG nova.wsgi [-] Loading app osapi_compute from /etc/nova/api-paste.ini load_app /usr/lib/python2.7/dist-packages/nova/wsgi.py:481
So, the change in api-paste.ini should be reloaded.

And, this is also happen for glance, and cinder.

Then, I let the wrong password stays in /etc/nova/api-paste.ini for a night(10,12 hours ?), then finally I get the error:
nova list
ERROR: Unauthorized (HTTP 401)

Looks like the change to api-paste.ini do not work directly if I was in a correct auth before.

[li,chen (chen-li)]

Hi dolph,

Why you think this is a question not a bug?
The change in api-paste.ini do not work directly after service reboot is by design?
Why?

[Dolph Mathews (dolph)]

Apologies, I did mean to respond earlier. It looks like you're using PKI tokens, which can be validated offline on the nova side. I'm not clear on whether the token was still valid or not, but the token revocation list would have been cached by nova before the token was revoked/expired, so I don't see any unexpected behavior here. Both the token's expiration, the frequency of calls to get token revocation lists can both be configured if the default values aren't suitable for your deployment.

[li,chen (chen-li)]

Thanks Dolph.

Can you more specific on "validated offline on the nova side" ?
Because I have cleaned all tokens in keystone database directly by hand "delete from token".
Although it is a rude behavior, but I think it will not affect services working, and it should clean all tokens no matter it is still valid or not.

And what is a "token revocation list" ? I have asked many places about this, but all I get is that a revoke token means "not expired but deleted by user".
And how can I change "the frequency of calls to get token revocation lists " ?
Because sometime revoke happened every time when I run commands, sometimes it just not happened.
This makes me really confusing.....

Thanks.
-chen

[Dolph Mathews (dolph)]

Deleting tokens is not the same as revoking them in the case of PKI.

In the case of UUID tokens, deleting tokens from keystone results in online validation failure, which is the behavior I think you're looking for. If that's the case, considering switching keystone.conf token_format to 'UUID'.

In the case of PKI tokens, they can be validated client-side (in the nova pipeline) without reaching out to keystone (where you're actually deleting token records... an action that has no consequence other than losing audit capabilities for PKI). Revoked PKI tokens are published in the token revocation list, which is periodically pulled by auth_token and results in otherwise valid (not expired) tokens being denied. For example, revoking a role from a user results in that user's otherwise valid tokens being revoked.

To explicitly revoke a specific token (regardless of PKI or UUID), use the API rather than modifying the database directly, e.g. DELETE /v2.0/tokens/{token_id}

[li,chen (chen-li)]

How "validated client-side (in the nova pipeline) without reaching out to keystone" ? I don't understand, If it not reach keystone, how nova know user's token is valid or not ? How nova make this decision ?

And, when would a revoke happen ?

[Dolph Mathews (dolph)]

Check out chmouel's blog: http://blog.chmouel.com/2013/05/02/keystone-pki-tokens-overview/