[OSSA 2013-011] Deleted user can still create instances
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
| OpenStack Identity (keystone) |
High
|
Dolph Mathews | ||
| Folsom |
High
|
Dolph Mathews | ||
| Grizzly |
High
|
Dolph Mathews | ||
| OpenStack Security Advisory |
Undecided
|
Thierry Carrez | ||
| keystone (Ubuntu) |
Undecided
|
Unassigned | ||
| Quantal |
Undecided
|
Unassigned | ||
| Raring |
Undecided
|
Unassigned |
Bug Description
Description:
A deleted user is still able to create instances and do other stuff if he's still logged in.
Steps to reproduce:
1. Login with admin user in Chrome
2. Login with demo user in Firefox
3. Use the admin user to delete the demo user
4. Go back to firefox and use the demo user to create an instance for example
Current result:
Demo user in firefox stays logged in and can create instances, but I guess he can do anything he want with his token
Expected result:
Demo user shouldn't be able to still create instances, or do other stuff. Instead he should be automatically logged out as soon as we notice that he's already deleted.
description: | updated |
Dolph Mathews (dolph) wrote : | #2 |
Deleting a user should result in all of the user's tokens being revoked, which should be recognized by the auth_token middleware (either by revocation list for PKI or by calling keystone per token for UUID), causing subsequent requests by the user to result in 401's.
Can you confirm:
- which type of tokens are in use, UUID or PKI?
- which middleware is protecting nova? keystone.
- how is the middleware configured? specifically interested in revocation_
- which release/branch of keystone is this occurring on?
Adam Young (ayoung) wrote : | #3 |
For PKI token (the default in Grizzly), there will be a time delay between the revocation and the fetch of the revocation list. This is expected and as designed. The time delay can be set as low as desired.
Changed in keystone: | |
status: | New → Incomplete |
Dolph Mathews (dolph) wrote : | #4 |
Also, I assume step 4 was meant to read "4. Go back to **firefox**..."
Sam Stoelinga (sammiestoel) wrote : | #5 |
"Also, I assume step 4 was meant to read "4. Go back to **firefox**...""
Yes a typo, have updated the description
- which type of tokens are in use, UUID or PKI?
select * from token LIMIT 0,1\G
*******
id: 0029fab4803b4ca
expires: 2013-04-12 02:28:14
extra: {"user": {"email": "<email address hidden>", "enabled": true, "id": "e81e1688ce7b4e
valid: 1
1 row in set (0.00 sec)
Not sure if this answers your question. If not could you tell me how to check this?
- which middleware is protecting nova? keystone.
[filter:authtoken]
paste.filter_
- how is the middleware configured? specifically interested in revocation_
Seems we don't have this set in nova.conf, api-paste.conf and keystone.conf. Any other place to check this? This should be a keystone option I guess?
- which release/branch of keystone is this occurring on?
dpkg -l | grep keystone
ii keystone 2012.2.
ii python-keystone 2012.2.
ii python-
Those are the folsom cloudarchive packages for ubuntu 12.04
I'm not sure if I answered your questions correclty, so please see related configurations below.
nova.conf:
http://
nova/api-
http://
keystone.conf
http://
description: | updated |
Dolph Mathews (dolph) wrote : | #6 |
Thanks, all that was helpful! So, UUID tokens owned by a deleted user in folsom are not denied by keystone.
Changed in keystone: | |
status: | Incomplete → New |
Thierry Carrez (ttx) wrote : | #7 |
Still trying to make up my mind whether to consider this a "vulnerability".
I suspect those tokens end up being denied after their validity period ? Is there a way for the admin to disable the tokens manually ?
Is that only affecting Folsom ?
Dolph Mathews (dolph) wrote : | #8 |
> Still trying to make up my mind whether to consider this a "vulnerability".
I'm leaning towards yes.
> I suspect those tokens end up being denied after their validity period?
I don't believe that option existing in Folsom - UUID tokens should be denied immediately.
> Is there a way for the admin to disable the tokens manually?
Yes, you can delete tokens one at a time if you know the token ( DELETE :35357/
My guess is that there is a difference in behavior between disabling a user and deleting a user (both should result in all associated tokens being revoked). As a workaround, I'd suggest disabling the user prior to deletion.
> Is that only affecting Folsom?
Probably, but grizzly needs to be tested as well.
Thierry Carrez (ttx) wrote : | #9 |
OK, we'll do staged disclosure on that one, so please do not post any patch publicly before the coordinated release date.
Attach patches to this (private) bug instead.
Changed in keystone: | |
importance: | Undecided → High |
status: | New → Confirmed |
description: | updated |
Thierry Carrez (ttx) wrote : | #10 |
Keystone core: feeling like producing a patch for this one ? Also confirming Grizzly being affected or not would be nice.
Dolph Mathews (dolph) wrote : | #11 |
I'll pick this up later this week if no one else has.
Adam Young (ayoung) wrote : | #12 |
UUID Tokens are authenticated via an online call to Keystone. Once that has been done,. they are cached inside auth_token middleware. The is is a performance optimization. In order to perform and additional lookup against Keystone, set the cache value lower. So the trade off is between a performance optimization and immediate notification of a revocation event. The solution to this is to shorten the time UUID tokens are in the cache.
We default this value in the auth_token middleware initialization.
cfg.
Thierry Carrez (ttx) wrote : | #13 |
@Adam: if I understand you correctly, we should not consider this a vulnerability, but this being designed behavior. This was my original thought, but Dolph's comments made me reconsider.
@Dolph: do you confirm Adam's point of view ? If yes, I'd just close this as Invalid.
Changed in keystone: | |
status: | Confirmed → Incomplete |
Dolph Mathews (dolph) wrote : | #14 |
I'm not clear on whether this should be impacted by token_cache_time (default=300 seconds) or revocation_
Either way, I'd like to attempt to reproduce with default settings and try turning down the token_cache_time before changing the status of the bug.
Dolph Mathews (dolph) wrote : | #15 |
I've been creating & destroying instances in horizon for 10 minutes after my user account was deleted (also via horizon).
Changed in keystone: | |
status: | Incomplete → Confirmed |
Dolph Mathews (dolph) wrote : | #16 |
Workaround: disabling the user prior to deletion causes 401's to be raised for the user almost immediately.
Dolph Mathews (dolph) wrote : | #17 |
immediately* after the token_cache_time has passed since the initial use of the token
Adam Young (ayoung) wrote : | #18 |
I see that delete_user is not in the Identity manager, but is rather implemented by the individual backends. I suspect that they are not correctly propagating a token revocation event upon deletion:
A quick brows of the code seems to validate this:
https:/
https:/
The controller call is
def delete_user(self, context, user_id):
Where as the call set_user_enabled calls update_user has a call self._delete_
It looks like the fix is to put code into the manager, roughly here:
https:/
That calls the underlying driver's delete_user code, and then deletes the tokens for the user.
Or add a call to
in delete_use
Adam Young (ayoung) wrote : | #19 |
Dolph Mathews (dolph) wrote : | #20 |
Fix & client test attached.
Dolph Mathews (dolph) wrote : | #21 |
Just noticed ayoung's patch above; we obviously agree on the solution :)
Adam Young (ayoung) wrote : | #22 |
+1 on Dolph's patch. TEsts are key, here.
Thierry Carrez (ttx) wrote : | #23 |
We need another keystone-core +1 on the patch
Could you produce a patch for stable/grizzly ? (and stable/folsom if that's affected as well)
About impact, let me make sure I understand this correctly: when users are deleted, their existing tokens are not invalidated, so they stay valid for the duration of the token life ? What is the default lifetime for a token ? Which types of tokens are affected ?
Henry Nash (henry-nash) wrote : | #24 |
+1 on the patch.
As an aside, we have this covered for the v3 api, it is just the v2 api that has the issue (i.e. the controller code that responds to the v3 delete user call already has the call to invalidate the tokens). So people calling the v3 api are not subject to this vulnerability.
Dolph Mathews (dolph) wrote : | #25 |
1) When users are deleted, their existing tokens are not invalidated, so they stay valid for the duration of the token life ? Yes, the workaround here is to disable the user prior to deleting them, which will result in tokens being revoked as expected.
2) What is the default lifetime for a token? 24 hours; this is configurable via keystone.conf [token] expiration (defaults to 86400 seconds)
3) Which types of tokens are affected? PKI and UUID, both.
And as henry-nash pointed out, this only affects the delete user call on the v2 API, but both APIs will continue to validate pre-existing tokens. After a user is either disabled or deleted, they are not able to generate new tokens.
Thierry Carrez (ttx) wrote : | #26 |
Here is proposed impact description, please doublecheck.
@Sam, do you want us to credit a specific company for the discovery, in addition to your name ?
=======
Title: Keystone tokens not immediately invalidated when user is deleted
Reporter: Sam Stoelinga
Products: Keystone
Affects: All versions
Description:
Sam Stoelinga reported a vulnerability in Keystone. When users are deleted through Keystone v2 API, existing tokens for those users are not immediately invalidated and remain valid for the duration of the token's life (by default, up to 24 hours). This may result in users retaining access when the administrator of the system thought them disabled. Keystone setups using the v3 API call to delete users are unaffected. You can workaround this issue by disabling a user before deleting it: in that case the tokens belonging to the disabled user are immediately invalidated.
=======
Henry Nash (henry-nash) wrote : | #28 |
Looks good - although maybe move the workaround description before the sentence where we say that v3 is unaffected - making it more obvious that you only need the workaround for v2.
Henry
On 29 Apr 2013, at 11:12, Thierry Carrez wrote:
> Here is proposed impact description, please doublecheck.
>
> @Sam, do you want us to credit a specific company for the discovery, in
> addition to your name ?
>
> =======
> Title: Keystone tokens not immediately invalidated when user is deleted
> Reporter: Sam Stoelinga
> Products: Keystone
> Affects: All versions
>
> Description:
> Sam Stoelinga reported a vulnerability in Keystone. When users are deleted through Keystone v2 API, existing tokens for those users are not immediately invalidated and remain valid for the duration of the token's life (by default, up to 24 hours). This may result in users retaining access when the administrator of the system thought them disabled. Keystone setups using the v3 API call to delete users are unaffected. You can workaround this issue by disabling a user before deleting it: in that case the tokens belonging to the disabled user are immediately invalidated.
> =======
>
> --
> You received this bug notification because you are a member of Keystone
> Core Developers, which is subscribed to the bug report.
> https:/
>
> Title:
> Deleted user can still create instances
>
> Status in OpenStack Identity (Keystone):
> Confirmed
>
> Bug description:
> Description:
> A deleted user is still able to create instances and do other stuff if he's still logged in.
>
> Steps to reproduce:
> 1. Login with admin user in Chrome
> 2. Login with demo user in Firefox
> 3. Use the admin user to delete the demo user
> 4. Go back to firefox and use the demo user to create an instance for example
>
> Current result:
> Demo user in firefox stays logged in and can create instances, but I guess he can do anything he want with his token
>
> Expected result:
> Demo user shouldn't be able to still create instances, or do other stuff. Instead he should be automatically logged out as soon as we notice that he's already deleted.
>
> To manage notifications about this bug go to:
> https:/
>
Sam Stoelinga (sammiestoel) wrote : Re: [Bug 1166670] Re: Deleted user can still create instances | #27 |
Seems my company still doesn't like working upstream so better not, just my
name is fine.
Thx
On Apr 29, 2013 6:20 PM, "Thierry Carrez" <email address hidden>
wrote:
> Here is proposed impact description, please doublecheck.
>
> @Sam, do you want us to credit a specific company for the discovery, in
> addition to your name ?
>
> =======
> Title: Keystone tokens not immediately invalidated when user is deleted
> Reporter: Sam Stoelinga
> Products: Keystone
> Affects: All versions
>
> Description:
> Sam Stoelinga reported a vulnerability in Keystone. When users are deleted
> through Keystone v2 API, existing tokens for those users are not
> immediately invalidated and remain valid for the duration of the token's
> life (by default, up to 24 hours). This may result in users retaining
> access when the administrator of the system thought them disabled. Keystone
> setups using the v3 API call to delete users are unaffected. You can
> workaround this issue by disabling a user before deleting it: in that case
> the tokens belonging to the disabled user are immediately invalidated.
> =======
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https:/
>
> Title:
> Deleted user can still create instances
>
> Status in OpenStack Identity (Keystone):
> Confirmed
>
> Bug description:
> Description:
> A deleted user is still able to create instances and do other stuff if
> he's still logged in.
>
> Steps to reproduce:
> 1. Login with admin user in Chrome
> 2. Login with demo user in Firefox
> 3. Use the admin user to delete the demo user
> 4. Go back to firefox and use the demo user to create an instance for
> example
>
> Current result:
> Demo user in firefox stays logged in and can create instances, but I
> guess he can do anything he want with his token
>
> Expected result:
> Demo user shouldn't be able to still create instances, or do other
> stuff. Instead he should be automatically logged out as soon as we notice
> that he's already deleted.
>
> To manage notifications about this bug go to:
> https:/
>
@Henry: Agreed -- see below. Another +1 on the impact description would be nice.
=======
Title: Keystone tokens not immediately invalidated when user is deleted
Reporter: Sam Stoelinga
Products: Keystone
Affects: All versions
Description:
Sam Stoelinga reported a vulnerability in Keystone. When users are deleted through Keystone v2 API, existing tokens for those users are not immediately invalidated and remain valid for the duration of the token's life (by default, up to 24 hours). This may result in users retaining access when the administrator of the system thought them disabled. You can workaround this issue by disabling a user before deleting it: in that case the tokens belonging to the disabled user are immediately invalidated. Keystone setups using the v3 API call to delete users are unaffected.
=======
Thierry Carrez (ttx) wrote : | #30 |
Also we need a Grizzly backport for the master patch (probably easy) and potentially a Folsom backport if Folsom is affected.
Sam Stoelinga (sammiestoel) wrote : Re: [Bug 1166670] Re: Deleted user can still create instances | #31 |
Folsom is also affected.
On Tue, Apr 30, 2013 at 7:52 PM, Thierry Carrez <<email address hidden>
> wrote:
> Also we need a Grizzly backport for the master patch (probably easy) and
> potentially a Folsom backport if Folsom is affected.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https:/
>
> Title:
> Deleted user can still create instances
>
> Status in OpenStack Identity (Keystone):
> Confirmed
>
> Bug description:
> Description:
> A deleted user is still able to create instances and do other stuff if
> he's still logged in.
>
> Steps to reproduce:
> 1. Login with admin user in Chrome
> 2. Login with demo user in Firefox
> 3. Use the admin user to delete the demo user
> 4. Go back to firefox and use the demo user to create an instance for
> example
>
> Current result:
> Demo user in firefox stays logged in and can create instances, but I
> guess he can do anything he want with his token
>
> Expected result:
> Demo user shouldn't be able to still create instances, or do other
> stuff. Instead he should be automatically logged out as soon as we notice
> that he's already deleted.
>
> To manage notifications about this bug go to:
> https:/
>
Apologies for the delay on backports!
Dolph Mathews (dolph) wrote : | #33 |
The fix in folsom is just a (bigger) copy/paste from update_user()
Adam Young (ayoung) wrote : | #34 |
+1 on grizzly back port.
+1 on folsom backport, but note that for the memcached backend, revocations are not supported.
Thierry Carrez (ttx) wrote : | #35 |
Could use another keystone-core +1 on the backports, then it's ready to go to downstream stakeholders.
Changed in keystone: | |
assignee: | nobody → Dolph Mathews (dolph) |
Brant Knudson (blk-u) wrote : | #36 |
I reviewed the 3 patches:
bug-1166670-master-v1
bug-1166670-grizzly-v1
bug-1166670-folsom-v1
and they all look like they fix the problem and are correct.
One nit, in folsom, I think the text should be:
rather than
Thierry Carrez (ttx) wrote : | #37 |
Sent to downstream stakeholders.
Proposed public disclosure date/time: Thursday, May 9, 1500UTC.
Changed in keystone: | |
status: | Confirmed → Triaged |
Thierry Carrez (ttx) wrote : | #38 |
This also affects Essex. Adding Thomas and the canonical security team to the bug so that they can synchronize on an Essex patch. Note that the code is very similar in Essex, so a similar patch should work.
Jamie Strandboge (jdstrand) wrote : | #39 |
Here is a lightly tested patch against Ubuntu's Essex based on Folsom's patch. There were no code changes, just changes to make it apply cleanly. It passes the keystone testsuite (including the newly added tests).
Thomas Goirand (thomas-goirand) wrote : | #40 |
Hi Jamie,
Thanks a lot for sharing your patch. Indeed, I will need to maintain Essex for as long as you guys will, so I will also work on security patches each time we will have an issue. Working with you will be very nice indeed, I probably can also produce some patch backports in the future.
FYI, your patch applies also on the Debian package.
Thomas
information type: | Private Security → Public Security |
Fix proposed to branch: master
Review: https:/
Changed in keystone: | |
status: | Triaged → In Progress |
Fix proposed to branch: stable/grizzly
Review: https:/
Fix proposed to branch: stable/folsom
Review: https:/
OSSA-2013-011
Reviewed: https:/
Committed: http://
Submitter: Jenkins
Branch: master
commit 33214f311aa36b1
Author: Dolph Mathews <email address hidden>
Date: Thu May 9 10:05:43 2013 -0500
Revoke tokens on user delete (bug 1166670)
Change-Id: Ieeb2246579ddc0
Changed in keystone: | |
status: | In Progress → Fix Committed |
Reviewed: https:/
Committed: http://
Submitter: Jenkins
Branch: stable/grizzly
commit 678b06a91f772d6
Author: Dolph Mathews <email address hidden>
Date: Thu May 9 10:06:35 2013 -0500
Revoke tokens on user delete (bug 1166670)
Change-Id: Id8a2972c9adb57
Reviewed: https:/
Committed: http://
Submitter: Jenkins
Branch: stable/folsom
commit 992466d1dbf80a9
Author: Dolph Mathews <email address hidden>
Date: Thu May 9 10:07:26 2013 -0500
Revoke tokens on user delete (bug 1166670)
Change-Id: I8dacf71b43cffc
summary: |
- Deleted user can still create instances + [OSSA 2013-011] Deleted user can still create instances |
Changed in ossa: | |
assignee: | nobody → Thierry Carrez (ttx) |
status: | New → Fix Released |
Changed in keystone: | |
milestone: | none → havana-1 |
status: | Fix Committed → Fix Released |
Hello Sam, or anyone else affected,
Accepted keystone into quantal-proposed. The package will build now and be available at http://
Please help us by testing this new package. See https:/
If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-
Further information regarding the verification process can be found at https:/
tags: | added: verification-needed |
Adam Gandelman (gandelman-a) wrote : | #49 |
Fixed Released in Ubuntu Raring: keystone 1:2013.1-0ubuntu1.1
FIxed Released in Ubuntu Quantal: keystone 2012.2.
Changed in ubuntu: | |
status: | New → Invalid |
no longer affects: | ubuntu |
Changed in keystone (Ubuntu): | |
status: | New → Invalid |
Changed in keystone (Ubuntu Quantal): | |
status: | New → Fix Released |
Changed in keystone (Ubuntu Raring): | |
status: | New → Fix Released |
tags: |
added: verification-done removed: verification-needed |
Adam Conrad (adconrad) wrote : Update Released | #50 |
The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.
Changed in keystone: | |
milestone: | havana-1 → 2013.2 |
Not really a Horizon issue, switching to Keystone.
I think this is by design how tokens work, but I will ask for keystone core confirmation first.