[OSSA 2013-011] Deleted user can still create instances

[Sam Stoelinga]

Description:
A deleted user is still able to create instances and do other stuff if he's still logged in.

Steps to reproduce:
1. Login with admin user in Chrome
2. Login with demo user in Firefox
3. Use the admin user to delete the demo user
4. Go back to firefox and use the demo user to create an instance for example

Current result:
Demo user in firefox stays logged in and can create instances, but I guess he can do anything he want with his token

Expected result:
Demo user shouldn't be able to still create instances, or do other stuff. Instead he should be automatically logged out as soon as we notice that he's already deleted.

[Thierry Carrez (ttx)]

Not really a Horizon issue, switching to Keystone.

I think this is by design how tokens work, but I will ask for keystone core confirmation first.

[Dolph Mathews (dolph)]

Deleting a user should result in all of the user's tokens being revoked, which should be recognized by the auth_token middleware (either by revocation list for PKI or by calling keystone per token for UUID), causing subsequent requests by the user to result in 401's.

Can you confirm:

- which type of tokens are in use, UUID or PKI?
- which middleware is protecting nova? keystone.middleware.auth_token or keystoneclient.middleware.auth_token?
- how is the middleware configured? specifically interested in revocation_cache_time if set, which should default to 1 second
- which release/branch of keystone is this occurring on?

[Adam Young (ayoung)]

For PKI token (the default in Grizzly), there will be a time delay between the revocation and the fetch of the revocation list. This is expected and as designed. The time delay can be set as low as desired.

[Dolph Mathews (dolph)]

Also, I assume step 4 was meant to read "4. Go back to **firefox**..."

[Sam Stoelinga (sammiestoel)]

"Also, I assume step 4 was meant to read "4. Go back to **firefox**...""
Yes a typo, have updated the description

- which type of tokens are in use, UUID or PKI?
select * from token LIMIT 0,1\G
*************************** 1. row ***************************
     id: 0029fab4803b4caab6b0e0d7b47fd608
expires: 2013-04-12 02:28:14
  extra: {"user": {"email": "<email address hidden>", "enabled": true, "id": "e81e1688ce7b4ee1a7af1cd12d084870", "name": "admin", "tenantId": null}, "key": "0029fab4803b4caab6b0e0d7b47fd608", "tenant": null, "metadata": {}}
  valid: 1
1 row in set (0.00 sec)

Not sure if this answers your question. If not could you tell me how to check this?

- which middleware is protecting nova? keystone.middleware.auth_token or keystoneclient.middleware.auth_token?
[filter:authtoken]
paste.filter_factory = keystone.middleware.auth_token:filter_factory

- how is the middleware configured? specifically interested in revocation_cache_time if set, which should default to 1 second
Seems we don't have this set in nova.conf, api-paste.conf and keystone.conf. Any other place to check this? This should be a keystone option I guess?

- which release/branch of keystone is this occurring on?
dpkg -l | grep keystone
ii keystone 2012.2.1-0ubuntu1~cloud0 OpenStack identity service - Daemons
ii python-keystone 2012.2.1-0ubuntu1~cloud0 OpenStack identity service - Python library
ii python-keystoneclient 1:0.1.3-0ubuntu1.1~cloud0 Client libary for Openstack Keystone API

Those are the folsom cloudarchive packages for ubuntu 12.04

I'm not sure if I answered your questions correclty, so please see related configurations below.

nova.conf:
http://paste.openstack.org/show/35770/

nova/api-paste.conf:
http://paste.openstack.org/show/35771/

keystone.conf
http://paste.openstack.org/show/35774/

[Dolph Mathews (dolph)]

Thanks, all that was helpful! So, UUID tokens owned by a deleted user in folsom are not denied by keystone.middleware.auth_token

[Thierry Carrez (ttx)]

Still trying to make up my mind whether to consider this a "vulnerability".

I suspect those tokens end up being denied after their validity period ? Is there a way for the admin to disable the tokens manually ?

Is that only affecting Folsom ?

[Dolph Mathews (dolph)]

> Still trying to make up my mind whether to consider this a "vulnerability".

I'm leaning towards yes.

> I suspect those tokens end up being denied after their validity period?

I don't believe that option existing in Folsom - UUID tokens should be denied immediately.

> Is there a way for the admin to disable the tokens manually?

Yes, you can delete tokens one at a time if you know the token ( DELETE :35357/v2.0/tokens/{token_id} )

My guess is that there is a difference in behavior between disabling a user and deleting a user (both should result in all associated tokens being revoked). As a workaround, I'd suggest disabling the user prior to deletion.

> Is that only affecting Folsom?

Probably, but grizzly needs to be tested as well.

[Thierry Carrez (ttx)]

OK, we'll do staged disclosure on that one, so please do not post any patch publicly before the coordinated release date.
Attach patches to this (private) bug instead.

[Thierry Carrez (ttx)]

Keystone core: feeling like producing a patch for this one ? Also confirming Grizzly being affected or not would be nice.

[Dolph Mathews (dolph)]

I'll pick this up later this week if no one else has.

[Adam Young (ayoung)]

UUID Tokens are authenticated via an online call to Keystone. Once that has been done,. they are cached inside auth_token middleware. The is is a performance optimization. In order to perform and additional lookup against Keystone, set the cache value lower. So the trade off is between a performance optimization and immediate notification of a revocation event. The solution to this is to shorten the time UUID tokens are in the cache.

We default this value in the auth_token middleware initialization.
    cfg.IntOpt('token_cache_time', default=300), With the time in seconds, 300 means five minutes. This seems like a reasonable trade-off. We can choose a shorter default. And installations can configure this value with the application paste files.

[Thierry Carrez (ttx)]

@Adam: if I understand you correctly, we should not consider this a vulnerability, but this being designed behavior. This was my original thought, but Dolph's comments made me reconsider.

@Dolph: do you confirm Adam's point of view ? If yes, I'd just close this as Invalid.

[Dolph Mathews (dolph)]

I'm not clear on whether this should be impacted by token_cache_time (default=300 seconds) or revocation_cache_time (default=1 second)

Either way, I'd like to attempt to reproduce with default settings and try turning down the token_cache_time before changing the status of the bug.

[Dolph Mathews (dolph)]

I've been creating & destroying instances in horizon for 10 minutes after my user account was deleted (also via horizon).

[Dolph Mathews (dolph)]

Workaround: disabling the user prior to deletion causes 401's to be raised for the user almost immediately.

[Dolph Mathews (dolph)]

immediately* after the token_cache_time has passed since the initial use of the token

[Adam Young (ayoung)]

I see that delete_user is not in the Identity manager, but is rather implemented by the individual backends. I suspect that they are not correctly propagating a token revocation event upon deletion:

A quick brows of the code seems to validate this:
https://github.com/openstack/keystone/blob/master/keystone/identity/backends/sql.py#L760
https://github.com/openstack/keystone/blob/master/keystone/identity/backends/ldap/core.py#L215

The controller call is
def delete_user(self, context, user_id):
        self.assert_admin(context)
        self.identity_api.delete_user(context, user_id)

Where as the call set_user_enabled calls update_user has a call self._delete_tokens_for_user(context, user_id)

It looks like the fix is to put code into the manager, roughly here:

 https://github.com/openstack/keystone/blob/master/keystone/identity/core.py#L84

That calls the underlying driver's delete_user code, and then deletes the tokens for the user.

Or add a call to
            self._delete_tokens_for_user(context, user_id)

in delete_use

[Dolph Mathews (dolph)]

Fix & client test attached.

[Dolph Mathews (dolph)]

Just noticed ayoung's patch above; we obviously agree on the solution :)

[Adam Young (ayoung)]

+1 on Dolph's patch. TEsts are key, here.

[Thierry Carrez (ttx)]

We need another keystone-core +1 on the patch

Could you produce a patch for stable/grizzly ? (and stable/folsom if that's affected as well)

About impact, let me make sure I understand this correctly: when users are deleted, their existing tokens are not invalidated, so they stay valid for the duration of the token life ? What is the default lifetime for a token ? Which types of tokens are affected ?

[Henry Nash (henry-nash)]

+1 on the patch.

As an aside, we have this covered for the v3 api, it is just the v2 api that has the issue (i.e. the controller code that responds to the v3 delete user call already has the call to invalidate the tokens). So people calling the v3 api are not subject to this vulnerability.

[Dolph Mathews (dolph)]

1) When users are deleted, their existing tokens are not invalidated, so they stay valid for the duration of the token life ? Yes, the workaround here is to disable the user prior to deleting them, which will result in tokens being revoked as expected.

2) What is the default lifetime for a token? 24 hours; this is configurable via keystone.conf [token] expiration (defaults to 86400 seconds)

3) Which types of tokens are affected? PKI and UUID, both.

And as henry-nash pointed out, this only affects the delete user call on the v2 API, but both APIs will continue to validate pre-existing tokens. After a user is either disabled or deleted, they are not able to generate new tokens.

[Thierry Carrez (ttx)]

Here is proposed impact description, please doublecheck.

@Sam, do you want us to credit a specific company for the discovery, in addition to your name ?

================================================
Title: Keystone tokens not immediately invalidated when user is deleted
Reporter: Sam Stoelinga
Products: Keystone
Affects: All versions

Description:
Sam Stoelinga reported a vulnerability in Keystone. When users are deleted through Keystone v2 API, existing tokens for those users are not immediately invalidated and remain valid for the duration of the token's life (by default, up to 24 hours). This may result in users retaining access when the administrator of the system thought them disabled. Keystone setups using the v3 API call to delete users are unaffected. You can workaround this issue by disabling a user before deleting it: in that case the tokens belonging to the disabled user are immediately invalidated.
================================================

[Henry Nash (henry-nash)]

Looks good - although maybe move the workaround description before the sentence where we say that v3 is unaffected - making it more obvious that you only need the workaround for v2.

Henry
On 29 Apr 2013, at 11:12, Thierry Carrez wrote:

> Here is proposed impact description, please doublecheck.
>
> @Sam, do you want us to credit a specific company for the discovery, in
> addition to your name ?
>
> ================================================
> Title: Keystone tokens not immediately invalidated when user is deleted
> Reporter: Sam Stoelinga
> Products: Keystone
> Affects: All versions
>
> Description:
> Sam Stoelinga reported a vulnerability in Keystone. When users are deleted through Keystone v2 API, existing tokens for those users are not immediately invalidated and remain valid for the duration of the token's life (by default, up to 24 hours). This may result in users retaining access when the administrator of the system thought them disabled. Keystone setups using the v3 API call to delete users are unaffected. You can workaround this issue by disabling a user before deleting it: in that case the tokens belonging to the disabled user are immediately invalidated.
> ================================================
>
> --
> You received this bug notification because you are a member of Keystone
> Core Developers, which is subscribed to the bug report.
> https://bugs.launchpad.net/bugs/1166670
>
> Title:
> Deleted user can still create instances
>
> Status in OpenStack Identity (Keystone):
> Confirmed
>
> Bug description:
> Description:
> A deleted user is still able to create instances and do other stuff if he's still logged in.
>
> Steps to reproduce:
> 1. Login with admin user in Chrome
> 2. Login with demo user in Firefox
> 3. Use the admin user to delete the demo user
> 4. Go back to firefox and use the demo user to create an instance for example
>
> Current result:
> Demo user in firefox stays logged in and can create instances, but I guess he can do anything he want with his token
>
> Expected result:
> Demo user shouldn't be able to still create instances, or do other stuff. Instead he should be automatically logged out as soon as we notice that he's already deleted.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/keystone/+bug/1166670/+subscriptions
>

[Sam Stoelinga (sammiestoel)]

Seems my company still doesn't like working upstream so better not, just my
name is fine.

Thx
On Apr 29, 2013 6:20 PM, "Thierry Carrez" <email address hidden>
wrote:

> Here is proposed impact description, please doublecheck.
>
> @Sam, do you want us to credit a specific company for the discovery, in
> addition to your name ?
>
> ================================================
> Title: Keystone tokens not immediately invalidated when user is deleted
> Reporter: Sam Stoelinga
> Products: Keystone
> Affects: All versions
>
> Description:
> Sam Stoelinga reported a vulnerability in Keystone. When users are deleted
> through Keystone v2 API, existing tokens for those users are not
> immediately invalidated and remain valid for the duration of the token's
> life (by default, up to 24 hours). This may result in users retaining
> access when the administrator of the system thought them disabled. Keystone
> setups using the v3 API call to delete users are unaffected. You can
> workaround this issue by disabling a user before deleting it: in that case
> the tokens belonging to the disabled user are immediately invalidated.
> ================================================
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1166670
>
> Title:
> Deleted user can still create instances
>
> Status in OpenStack Identity (Keystone):
> Confirmed
>
> Bug description:
> Description:
> A deleted user is still able to create instances and do other stuff if
> he's still logged in.
>
> Steps to reproduce:
> 1. Login with admin user in Chrome
> 2. Login with demo user in Firefox
> 3. Use the admin user to delete the demo user
> 4. Go back to firefox and use the demo user to create an instance for
> example
>
> Current result:
> Demo user in firefox stays logged in and can create instances, but I
> guess he can do anything he want with his token
>
> Expected result:
> Demo user shouldn't be able to still create instances, or do other
> stuff. Instead he should be automatically logged out as soon as we notice
> that he's already deleted.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/keystone/+bug/1166670/+subscriptions
>

[Thierry Carrez (ttx)]

@Henry: Agreed -- see below. Another +1 on the impact description would be nice.

================================================
Title: Keystone tokens not immediately invalidated when user is deleted
Reporter: Sam Stoelinga
Products: Keystone
Affects: All versions

Description:
Sam Stoelinga reported a vulnerability in Keystone. When users are deleted through Keystone v2 API, existing tokens for those users are not immediately invalidated and remain valid for the duration of the token's life (by default, up to 24 hours). This may result in users retaining access when the administrator of the system thought them disabled. You can workaround this issue by disabling a user before deleting it: in that case the tokens belonging to the disabled user are immediately invalidated. Keystone setups using the v3 API call to delete users are unaffected.
================================================

[Thierry Carrez (ttx)]

Also we need a Grizzly backport for the master patch (probably easy) and potentially a Folsom backport if Folsom is affected.

[Sam Stoelinga (sammiestoel)]

Folsom is also affected.

On Tue, Apr 30, 2013 at 7:52 PM, Thierry Carrez <<email address hidden>
> wrote:

> Also we need a Grizzly backport for the master patch (probably easy) and
> potentially a Folsom backport if Folsom is affected.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1166670
>
> Title:
> Deleted user can still create instances
>
> Status in OpenStack Identity (Keystone):
> Confirmed
>
> Bug description:
> Description:
> A deleted user is still able to create instances and do other stuff if
> he's still logged in.
>
> Steps to reproduce:
> 1. Login with admin user in Chrome
> 2. Login with demo user in Firefox
> 3. Use the admin user to delete the demo user
> 4. Go back to firefox and use the demo user to create an instance for
> example
>
> Current result:
> Demo user in firefox stays logged in and can create instances, but I
> guess he can do anything he want with his token
>
> Expected result:
> Demo user shouldn't be able to still create instances, or do other
> stuff. Instead he should be automatically logged out as soon as we notice
> that he's already deleted.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/keystone/+bug/1166670/+subscriptions
>

[Dolph Mathews (dolph)]

Apologies for the delay on backports!

[Dolph Mathews (dolph)]

The fix in folsom is just a (bigger) copy/paste from update_user()

[Adam Young (ayoung)]

+1 on grizzly back port.
+1 on folsom backport, but note that for the memcached backend, revocations are not supported.

[Thierry Carrez (ttx)]

Could use another keystone-core +1 on the backports, then it's ready to go to downstream stakeholders.

[Brant Knudson (blk-u)]

I reviewed the 3 patches:
 bug-1166670-master-v1
 bug-1166670-grizzly-v1
 bug-1166670-folsom-v1

and they all look like they fix the problem and are correct.

One nit, in folsom, I think the text should be:
            LOG.warning('User %s has been deleted, but existing tokens '
                        'remain valid' % user_id)
rather than
            LOG.warning('User %s status has changed, but existing tokens '
                        'remain valid' % user_id)

[Thierry Carrez (ttx)]

Sent to downstream stakeholders.
Proposed public disclosure date/time: Thursday, May 9, 1500UTC.

[Thierry Carrez (ttx)]

This also affects Essex. Adding Thomas and the canonical security team to the bug so that they can synchronize on an Essex patch. Note that the code is very similar in Essex, so a similar patch should work.

[Jamie Strandboge (jdstrand)]

Here is a lightly tested patch against Ubuntu's Essex based on Folsom's patch. There were no code changes, just changes to make it apply cleanly. It passes the keystone testsuite (including the newly added tests).

[Thomas Goirand (thomas-goirand)]

Hi Jamie,

Thanks a lot for sharing your patch. Indeed, I will need to maintain Essex for as long as you guys will, so I will also work on security patches each time we will have an issue. Working with you will be very nice indeed, I probably can also produce some patch backports in the future.

FYI, your patch applies also on the Debian package.

Thomas

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/28677

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/28678

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/folsom
Review: https://review.openstack.org/28679

[Thierry Carrez (ttx)]

OSSA-2013-011

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/28677
Committed: http://github.com/openstack/keystone/commit/33214f311aa36b17f8f5ff06bee2130bf061df8f
Submitter: Jenkins
Branch: master

commit 33214f311aa36b17f8f5ff06bee2130bf061df8f
Author: Dolph Mathews <email address hidden>
Date: Thu May 9 10:05:43 2013 -0500

    Revoke tokens on user delete (bug 1166670)

    Change-Id: Ieeb2246579ddc0608092eed5239258d2b4dbf42b

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/28678
Committed: http://github.com/openstack/keystone/commit/678b06a91f772d6be82eb54ed11f27e20f446b57
Submitter: Jenkins
Branch: stable/grizzly

commit 678b06a91f772d6be82eb54ed11f27e20f446b57
Author: Dolph Mathews <email address hidden>
Date: Thu May 9 10:06:35 2013 -0500

    Revoke tokens on user delete (bug 1166670)

    Change-Id: Id8a2972c9adb572ef9d2b7f03bcf07a8de5614ab

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/28679
Committed: http://github.com/openstack/keystone/commit/992466d1dbf80a940190703dedf800d6d12dede8
Submitter: Jenkins
Branch: stable/folsom

commit 992466d1dbf80a940190703dedf800d6d12dede8
Author: Dolph Mathews <email address hidden>
Date: Thu May 9 10:07:26 2013 -0500

    Revoke tokens on user delete (bug 1166670)

    Change-Id: I8dacf71b43cffc12439520e05918729e385cb60a

[Adam Conrad (adconrad)]

Hello Sam, or anyone else affected,

Accepted keystone into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/keystone/2012.2.4-0ubuntu3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Adam Gandelman (gandelman-a)]

Fixed Released in Ubuntu Raring: keystone 1:2013.1-0ubuntu1.1
FIxed Released in Ubuntu Quantal: keystone 2012.2.3+stable-20130206-82c87e56-0ubuntu2.1

[Adam Conrad (adconrad)]

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.