[OSSA-2012-018] Removing user from a tenant isn't invalidating user access to tenant

Bug #1064914 reported by Vijaya Erukala
262
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
Fix Released
Critical
Vish Ishaya
Essex
Fix Released
Undecided
Yolanda Robla
Folsom
Fix Released
Critical
Thierry Carrez
OpenStack Security Advisory
Fix Released
Undecided
Thierry Carrez
keystone (Ubuntu)
Fix Released
Undecided
Unassigned
Nominated for Precise by Yolanda Robla
Quantal
Fix Released
Undecided
Unassigned

Bug Description

Was: (Able to access ec2 resources with out a user-role)

Steps to reproduce:

1.I have created a user,tenant and ec2 credentials using keystone and didnt associated a role
then try to run
$ euca-describe-images
Unauthorized: Failure communicating with keystone2

2.then associated a role to the user
$keystone user-role-add --user-id ee32bff3e1524a2b82fe804aac0ce682 --tenant-id cf38a72df8a14bb8984c699edfceb1c3 --role-id 26760c0f9ed045fcb70112f7b16813b3

3.then again I ran
$ euca-describe-images
IMAGE ami-00000002 None (cirros-0.3.0-x86_64-uec) 597b866b450d491f889b9432a564e9a9 available public machineaki-00000001 ari-00000003 instance-store
IMAGE ari-00000003 None (cirros-0.3.0-x86_64-uec-ramdisk) 597b866b450d491f889b9432a564e9a9 available public ramdisk instance-store
IMAGE aki-00000001 None (cirros-0.3.0-x86_64-uec-kernel) 597b866b450d491f889b9432a564e9a9 available public kernel instance-store

4.remove the user-role
keystone user-role-delete --user-id ee32bff3e1524a2b82fe804aac0ce682 --tenant-id cf38a72df8a14bb8984c699edfceb1c3 --role-id 26760c0f9ed045fcb70112f7b16813b3

5.now the user doesn't have any role associated,now again run
 $ euca-describe-images
IMAGE ami-00000002 None (cirros-0.3.0-x86_64-uec) 597b866b450d491f889b9432a564e9a9 available public machineaki-00000001 ari-00000003 instance-store
IMAGE ari-00000003 None (cirros-0.3.0-x86_64-uec-ramdisk) 597b866b450d491f889b9432a564e9a9 available public ramdisk instance-store
IMAGE aki-00000001 None (cirros-0.3.0-x86_64-uec-kernel) 597b866b450d491f889b9432a564e9a9 available public kernel instance-store

here user should get the message as in step 2 but it was successful.

not sure if this is a bug with keystone or ec2.

tags: added: ec2
Revision history for this message
Russell Bryant (russellb) wrote :

I have subscribed the vulnerability management team to this bug. We need to evaluate this as a potential security vulnerability.

information type: Public → Public Security
affects: nova → keystone
Changed in keystone:
importance: Undecided → Critical
status: New → Triaged
Revision history for this message
Thierry Carrez (ttx) wrote :

If confirmed, it's in the "extension of validity" class like the ones where you could extend the lifetime of a token. A bit hard to exploit (since you need to be given the role in the first place) but definitely needs to be fixed. I would issue an OSSA on this for the same reason we issued one on the previous token issues.

Revision history for this message
Joseph Heck (heckj) wrote :

The EC2 mechanisms disregard role entirely, and work from the relationship of the user to the tenant. If that relationship didn't exist, the user shouldn't be able to access the tenant's resources through EC2. WIth the V2 API, just associating a user with a tenant is an implicit "membership" role - and in fact that's how Horizon displays this information.

Actual "roles" as Keystone defines them are irrelevant to the EC2 credential access, and with the V2 API's and the EC2 mechanisms, roles have no function there. They're not checked as a form of RBAC for allowing EC2 credential access.

The V3 API is aiming to clean this up a bit, more for the OpenStack API. V3 API changes does not currently specify any RBAC around EC2 credential authority.

Changed in keystone:
status: Triaged → Invalid
Thierry Carrez (ttx)
information type: Public Security → Public
Revision history for this message
Vish Ishaya (vishvananda) wrote :

We don't check that the user is still a member of the tenant (project) so the ec2 (or s3) token still works

Changed in keystone:
status: Invalid → Confirmed
Revision history for this message
Joseph Heck (heckj) wrote :

vishy: heckj: correct, we don't check anywhere that the user is still a member of the tenant
[2:29pm] heckj: vishy: so the core of that bug is that we're not verifying that the user is a member of the tenant?
[2:30pm] vishy: correct
[2:30pm] vishy: i just marked it confirmed
[2:30pm] heckj: heh - I just marked it as triaged
[2:30pm] vishy: i just tested and removed myself from my tenant, I was still able to run commands with my ec2 token
[2:30pm] heckj: Okay - thank you! I'll get on it!

DELETE /tenants/{tenant_id}/users/{user_id}/roles/OS-KSADM/{role_id}
in keystone/contrib/admin_crud/core.py

Changed in keystone:
status: Confirmed → Triaged
status: Triaged → Confirmed
Revision history for this message
Joseph Heck (heckj) wrote :

[2:31pm] vishy: i can still do stuff through the nova api as well!

summary: - Able to access ec2 resources with out a user-role
+ Removing user from a tenant isn't invalidating user access to tenant
description: updated
Revision history for this message
Joseph Heck (heckj) wrote : Re: Removing user from a tenant isn't invalidating user access to tenant

related bug around removing a user from a tenant:
https://bugs.launchpad.net/python-keystoneclient/+bug/1078497

Changed in keystone:
status: Confirmed → Invalid
Revision history for this message
Vish Ishaya (vishvananda) wrote :

ok tracked this down and verified. The other bug above is separate, but this one still occurs. I managed to successfuly delete the role and the ec2 commands still work. This is because in the ec2 credentials we are storing the user and tenant. We should fix this by adding a check in the ec2 authenticate to make sure that the user still has a valid role before validating.

Changed in keystone:
status: Invalid → In Progress
assignee: nobody → Vish Ishaya (vishvananda)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (master)

Fix proposed to branch: master
Review: https://review.openstack.org/16028

tags: added: folsom-backport
Revision history for this message
Thierry Carrez (ttx) wrote : Re: Removing user from a tenant isn't invalidating user access to tenant

Looks like security is back on track !

information type: Public → Public Security
Revision history for this message
Thierry Carrez (ttx) wrote :

So if I get this straight, the issue is that if a user has a role that makes it associated with a tenant, but then that role is removed, the token is still valid. That would make it part of the "extension of validity" class I mentioned above, which we considered in the past as worthy of an advisory.

If confirmed, I'll prepare an impact statement.

Vish's folsom-backport tag probably means that Folsom is also affected. What about Essex at first glance ?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (master)

Reviewed: https://review.openstack.org/16028
Committed: http://github.com/openstack/keystone/commit/9d68b40cb9ea818c48152e6c712ff41586ad9653
Submitter: Jenkins
Branch: master

commit 9d68b40cb9ea818c48152e6c712ff41586ad9653
Author: Vishvananda Ishaya <email address hidden>
Date: Tue Nov 13 15:49:19 2012 -0800

    Ensures User is member of tenant in ec2 validation

    It is possible that a user is no longer a member of a tenant when
    they attempt to use an ec2 token. This checks to make sure that
    the user still has at least one valid role in the tenant before
    authenticating them. This should automatically work for the s3
    version as well since it is a subclass.

    Fixes bug 1064914

    Change-Id: Ieb237bae936a7b00ce7ba4d4c59aec6c7a69ec21

Changed in keystone:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (stable/folsom)

Fix proposed to branch: stable/folsom
Review: https://review.openstack.org/16304

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (stable/folsom)

Reviewed: https://review.openstack.org/16304
Committed: http://github.com/openstack/keystone/commit/37308dd4f3e33f7bd0f71d83fd51734d1870713b
Submitter: Jenkins
Branch: stable/folsom

commit 37308dd4f3e33f7bd0f71d83fd51734d1870713b
Author: Vishvananda Ishaya <email address hidden>
Date: Tue Nov 13 15:49:19 2012 -0800

    Ensures User is member of tenant in ec2 validation

    It is possible that a user is no longer a member of a tenant when
    they attempt to use an ec2 token. This checks to make sure that
    the user still has at least one valid role in the tenant before
    authenticating them. This should automatically work for the s3
    version as well since it is a subclass.

    Fixes bug 1064914

    Change-Id: Ieb237bae936a7b00ce7ba4d4c59aec6c7a69ec21

Revision history for this message
Thierry Carrez (ttx) wrote : Re: Removing user from a tenant isn't invalidating user access to tenant

@Russell: we have a precedent for issuing after-the-fact OSSA on this kind of thing. Do you think we should here ? If yes, maybe we can give a few days warning to the stakeholders while we work on the impact statement ?

Joseph Heck (heckj)
Changed in keystone:
milestone: none → grizzly-1
Revision history for this message
Russell Bryant (russellb) wrote :

Yes, I think based on our precedent, we should issue an OSSA for this.

Thierry Carrez (ttx)
Changed in keystone:
status: Fix Committed → Fix Released
Revision history for this message
Thierry Carrez (ttx) wrote :

Would be good to push an Essex backport for this as well -- CCing stable/essex team

Proposed impact description:

Title: EC2-style credentials invalidation issue
Reporter: Vijaya Erukala
Products: Keystone
Affects: All versions

Description:
Vijaya Erukala reported a vulnerability in Keystone EC2-style credentials invalidation: when a user is removed from a tenant, issued EC2-style credentials would continue to be valid for that tenant. An authenticated and authorized user could potentially leverage this vulnerability to extend his access beyond the account owner expectations. Only setups enabling EC2-style credentials (for example enabling EC2 API in Nova) are affected.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (stable/essex)

Fix proposed to branch: stable/essex
Review: https://review.openstack.org/16759

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: stable/essex
Review: https://review.openstack.org/16760

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (stable/essex)

Reviewed: https://review.openstack.org/16760
Committed: http://github.com/openstack/keystone/commit/8735009dc5b895db265a1cd573f39f4acfca2a19
Submitter: Jenkins
Branch: stable/essex

commit 8735009dc5b895db265a1cd573f39f4acfca2a19
Author: Vishvananda Ishaya <email address hidden>
Date: Tue Nov 13 15:49:19 2012 -0800

    Ensures User is member of tenant in ec2 validation

    It is possible that a user is no longer a member of a tenant when
    they attempt to use an ec2 token. This checks to make sure that
    the user still has at least one valid role in the tenant before
    authenticating them. This should automatically work for the s3
    version as well since it is a subclass.

    Fixes bug 1064914

    Change-Id: Ieb237bae936a7b00ce7ba4d4c59aec6c7a69ec21

Revision history for this message
Russell Bryant (russellb) wrote : Re: Removing user from a tenant isn't invalidating user access to tenant

@ttx: The vulnerability description sounds good to me. Thanks!

Revision history for this message
Thierry Carrez (ttx) wrote :

Pushed to downstream stakeholders. CVE-2012-5571

Revision history for this message
Thierry Carrez (ttx) wrote :

OSSA 2012-018

Changed in keystone (Ubuntu):
status: New → Fix Released
Changed in keystone (Ubuntu Quantal):
status: New → Confirmed
Revision history for this message
Jamie Strandboge (jdstrand) wrote :
Changed in keystone (Ubuntu Quantal):
status: Confirmed → Fix Released
Revision history for this message
Clint Byrum (clint-fewbar) wrote : Please test proposed package

Hello Vijaya, or anyone else affected,

Accepted keystone into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/keystone/2012.2.1-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-needed
Revision history for this message
Vijaya Erukala (vijaya-erukala) wrote : Re: Removing user from a tenant isn't invalidating user access to tenant

I have verified this on grizzly-2 and validation seems to be working fine.

$ keystone user-role-remove --user bd9c28b6e2794574b030ad5c3a7e4818 --tenant 97c5778a10f34b499a5abe1e2ecff24c --role 9fe2ff9ee4384b1894a90878d3e92bab

$ euca-describe-instances
Unauthorized: Failure communicating with keystone

Thierry Carrez (ttx)
Changed in keystone:
milestone: grizzly-1 → 2013.1
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Vijaya, or anyone else affected,

Accepted keystone into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/keystone/2012.1.3+stable-20130423-f48dd0fc-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Revision history for this message
Yolanda Robla (yolanda.robla) wrote : Verification report.

Please find the attached test log from the Ubuntu Server Team's CI infrastructure. As part of the verification process for this bug, Keystone has been deployed and configured across multiple nodes using precise-proposed as an installation source. After successful bring-up and configuration of the cluster, a number of exercises and smoke tests have be invoked to ensure the updated package did not introduce any regressions. A number of test iterations were carried out to catch any possible transient errors.

Please Note the list of installed packages at the top and bottom of the report.

For records of upstream test coverage of this update, please see the Jenkins links in the comments of the relevant upstream code-review(s):

Trunk review: https://review.openstack.org/16028
Stable review: https://review.openstack.org/16760

As per the provisional Micro Release Exception granted to this package by the Technical Board, we hope this contributes toward verification of this update.

Revision history for this message
Yolanda Robla (yolanda.robla) wrote : Re: Removing user from a tenant isn't invalidating user access to tenant

Test coverage log.

tags: added: verification-done
removed: verification-needed
Revision history for this message
Scott Kitterman (kitterman) wrote : Update Released

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Thierry Carrez (ttx)
summary: - Removing user from a tenant isn't invalidating user access to tenant
+ [OSSA-2012-018] Removing user from a tenant isn't invalidating user
+ access to tenant
Changed in ossa:
assignee: nobody → Thierry Carrez (ttx)
status: New → Fix Released
To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.