[0SSA 2014-011] RBAC policy not enforced when adding a security group rule using EC2 API (CVE-2014-0167)

Bug #1290537 reported by Marc Heckmann
262
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
High
Andrew Laski
Havana
High
Andrew Laski
OpenStack Security Advisory
Medium
Tristan Cacqueray

Bug Description

It seems that when using the EC2 API, the security group implementation does not enforce RBAC policy for the add_rules, remove_rules, destroy and other functions (in compute/api.py). Only the add_to_instance and remove_from_instance functions enforce RBAC. This seems like an oversight for obvious reasons.

The Nova API security group implementation does enforce RBAC on these functions.

In addition, the add_to_instance and remove_from _instance functions which are wrapped in RBAC verification use the "compute:security_groups" action which is not even listed in the default /etc/nova/policy.json. The latter is confusing to users.

This is the case on Grizlly and at first glance, it doesn't look like this has changed in Havana.

CVE References

Jeremy Stanley (fungi)
Changed in ossa:
status: New → Incomplete
Revision history for this message
Marc Heckmann (marc-w-heckmann) wrote :

What can I do to make this report more "complete"?

Thanks.

Revision history for this message
Jeremy Stanley (fungi) wrote :

I've marked the OpenStack Security Advisory task for this bug incomplete because the vulnerability management team is waiting for Nova core security reviewers to triage/confirm/reproduce the issue and determine a probable fix. Once we get closer to having a good grasp of the details there, the OSSA task will begin to separately progress through impact description drafting and review, CVE request and assignment, downstream notification and finally public disclosure (hopefully) synchronized with publication of the finalized patches. Hopefully that answers your questions.

Revision history for this message
Marc Heckmann (marc-w-heckmann) wrote :

Yes that answers my question. Thanks.

Let me know if there's anything I can do in the meantime.

Andrew Laski (alaski)
Changed in nova:
status: New → Confirmed
Revision history for this message
Andrew Laski (alaski) wrote :

I've confirmed the behaviour, and it seems that it's likely been this way for a while.

The trend in the OpenStack API has been to move policy out of the compute api and into the API itself. Even the existing checks on add_to_instance/remove_from_instance are likely to go away at some point. So to keep in line with that the fix for this would be to add policy checks into the ec2 API.

The code for policy enforcement is non invasive and easily backportable but we'll want to set a default policy that won't cause a problem for deployments. As mentioned by Marc, the current "compute:security_groups" rule isn't listed which means it uses the default rule. The default for the default rule is to limit calls to admins or users who share a project_id with the security group. If we set up new policies for the security group calls that use the default rule it may break a use case that relies on the unrestricted behaviour currently available, but I believe that's ok. And using the default rule will put the new policies in line with the existing policies for security groups calls.

Revision history for this message
Thierry Carrez (ttx) wrote :

Trying to make sure this is an exploitable vulnerability...

if add_to_instance and remove_from_instance are protected by RBAC, can you actually do any harm using the unprotected functions ? What would be the attack scenario ? Would for example using remove_rules remove securitygroup rules from instances ?

Revision history for this message
Marc Heckmann (marc-w-heckmann) wrote :

Assuming the default case of deny all in a security group, the attack scenario would be to use add_rules to add rules to an instance.

Now, the problem with this bug is that users may get into a situation where there is a false sense of security.

For example, as a private-cloud operator, I want to restrict all Nova actions to users with the "Member" role on the tenant. Once that is done, I want to create a new role that allows only for read-only access to security group and instance information for auditing purposes. I can achieve this by editing the rule for the "compute_extension:security_groups" action in /etc/nova/policy.json. This will do what I want for the Nova API. The problem is that the EC2 API is left vulnerable and a user that does not have the Member role but only the read-only role can now add or delete rules from a security group.

I hope that helps to clarify things. Of course, this bug only affects users who are trying to create tighter access controls. The default configuration is not affected by this bug since it grants any role on a project/tenant full access.

Revision history for this message
Thierry Carrez (ttx) wrote :

OK, that makes sense, thanks for the explanation! We'll issue an advisory for this one.
Andrew: would you be up for a patch ?

Changed in ossa:
importance: Undecided → Medium
status: Incomplete → Confirmed
Revision history for this message
Andrew Laski (alaski) wrote :

Thierry: yes, I can put a patch together for this.

Revision history for this message
Andrew Laski (alaski) wrote :

Here is an initial patch for review.

Revision history for this message
Thierry Carrez (ttx) wrote :

nova-coresec: care to review that one ?

Revision history for this message
Thierry Carrez (ttx) wrote :

A couple questions while I try to draft an impact description for the CVE:

* Mark, should we credit the company you work for, in addition to your name ? If yes, which company is it ?

* Any idea which versions of OpenStack should be considered vulnerable to this ? I suspect that would be when RBAC was added to openStack API, but not sure how to be more precise than that

Revision history for this message
Marc Heckmann (marc-w-heckmann) wrote :

I will check with my company and get back to you about the divulging the name.

For the versions, it sounds like this might go back pretty far. I can confirm that it affects Grizzly.

Revision history for this message
Marc Heckmann (marc-w-heckmann) wrote :

Hi, you can use the company name: Ubisoft

Jeremy Stanley (fungi)
Changed in nova:
assignee: nobody → Andrew Laski (alaski)
Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Impact description draft #1:

Title: RBAC policy not enforced in Nova EC2 API
Reporter: Marc Heckmann (Ubisoft)
Products: Nova
Versions: 2013.1 versions up to 2013.2.2

Description:
Marc Heckmann from Ubisoft reported a vulnerability in the Nova EC2 API security group implementation. RBAC policy are not enforced when using the EC2 API, in particular the add_rules, remove_rules and destroy methods. A restricted user may overcome his limitation by using EC2 API resulting in unauthorized action on security groups. Only setups using tighter access controls for Nova API are affected.

Changed in ossa:
assignee: nobody → Tristan Cacqueray (tristan-cacqueray)
Revision history for this message
Thierry Carrez (ttx) wrote :

Suggestions:

Title: "RBAC policy not properly enforced in Nova EC2 API"

RBAC policy are not enforced -> "policies"

and replace last sentence: "Only setups using non-default RBAC rules for Nova may be affected".

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

@ttx Thanks!

Impact description draft #2:

Title: RBAC policy not properly enforced in Nova EC2 API
Reporter: Marc Heckmann (Ubisoft)
Products: Nova
Versions: 2013.1 versions up to 2013.2.2

Description:
Marc Heckmann from Ubisoft reported a vulnerability in the Nova EC2 API security group implementation. RBAC policies are not enforced when using the EC2 API, in particular the add_rules, remove_rules and destroy methods. A restricted user may overcome his limitation by using EC2 API resulting in unauthorized action on security groups. Only setups using non-default RBAC rules for Nova may be affected.

Revision history for this message
Thierry Carrez (ttx) wrote :

+1 on Impact description draft #2

Thierry Carrez (ttx)
tags: added: icehouse-rc-potential
Revision history for this message
Russell Bryant (russellb) wrote :

+2 on the patch.

I'm going to add cyeoh to this bug, as I think he'd be a good nova-core member to help review this fix.

Thierry Carrez (ttx)
Changed in ossa:
status: Confirmed → Triaged
Revision history for this message
Jeremy Stanley (fungi) wrote :

Tristan's impact description in comment #16 looks good to me.

Revision history for this message
Christopher Yeoh (cyeoh-0) wrote :

+1 on the impact description

Re: the patch - Although I agree it fixes the problem I'm not sure its the way we should do it.

We already have a policy "compute_extension:security_groups" which is used by the Nova API to enforce
access in a general way to all of these features. So I'm thinking that rather than introducing new policies
more specific such as compute:security_groups:remove_rules settings to fix the bug we should just use
the existing generic compute_extension:security_groups rule in the ec2 API.

Now I think there may be a reasonable argument for having per method type policy settings in the API
rather than general ones and we have generally been moving this way anyway, but if we do so we need to
be consistent across the Nova and EC2 APIs. So if a compute:security_groups:remove_rules policy is used
by the EC2 code it should also be checked by the Nova API code too. Otherwise deployers could be mislead into
thinking the policy is affecting the Nova API when it isn't.

I think unless there is a clear need now to have detailed policy checks I think for an immediate bug fix we
should just use the existing general security groups policy (which I think also means no policy file changes
are needed by deployers just the code update). And then if we want to have more fine grained policy checks
we can do this in Juno. And this needs to be done fairly carefully for the Nova API side to ensure minimal
upgrade impact.

And more generally we definitely need to ensure all possibly policy checks are in the policy file rather than
rely on defaults as deployers should not be expected to grep the code to see what switches are available :-) I will
submit a blueprint for that work in Juno.

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

@Andrew Laski, Could you please attach another patch for Havana ?
The current one in comment #9 does apply to stable/havana but tests are failing, I think we are missing some import:

NameError: global name 'extensions' is not defined in nova/api/ec2/cloud.py
NameError: global name 'common_policy' is not defined in nova/tests/

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

But then extensions module is lacking check_compute_policy method.

summary: RBAC policy not enforced when adding a security group rule using EC2 API
+ (CVE-2014-0167)
Revision history for this message
John Garbutt (johngarbutt) wrote : Re: RBAC policy not enforced when adding a security group rule using EC2 API (CVE-2014-0167)

Yeah, I am -1 for what chris said.

Seems like we should use the same policy as the core openstack API, otherwise people will have to configure new policy for an api they may not release that had turned on, or something like that.

I agree that we should have a more granular policy on both in the long term, but thats not really the fix for this bug, as I see it.

Revision history for this message
Marc Heckmann (marc-w-heckmann) wrote :

I was hesitant to say anything about the patch, but Christopher is right, keeping the previously hidden "compute:security_groups" policy doesn't make sense if people are already using "compute_extension:security_groups". We should just have one way to do this.

And yeah, as an operator, having more fine grained policy is a plus, but this could be done in a future release.

Revision history for this message
Andrew Laski (alaski) wrote :

Christopher: Works for me. I'm happy with a simpler solution for now if everyone agrees. I'll update shortly.
And Tristan, I'll work on a backport but I just wanted to get approval on the direction first.

I added the fine grained controls into the ec2 api because the OpenStack api has fine grained controls as well, they're just enforced at the compute api level. Actually I see now that I should have just added the @wrap_check_security_groups_policy decorator for the compute api methods that the ec2 api uses. But really the ec2 api and OpenStack api have different semantics, and make different calls, so I think it's fine for them to have different policies. That can be handled later in a non security review though.

Revision history for this message
Andrew Laski (alaski) wrote :

Patch for master

Revision history for this message
Andrew Laski (alaski) wrote :

stable/havana

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Thank you Andrew, patches are passing tests on Havana and master (beside a seemingly unrelated test: nova.tests.virt.libvirt.test_libvirt.LibvirtConnTestCase.test_get_domain_info_with_more_return)

If core could approve it today I'll send the pre-OSSA with a disclosure date at:
2014-04-09 15:00 UTC

Thank you in advance!

Revision history for this message
Christopher Yeoh (cyeoh-0) wrote :

I'm +2 on the latest set of havana/master patches

Changed in ossa:
status: Triaged → In Progress
Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

the pre-OSSA have been sent.

Proposed public disclosure date/time:
2014-04-09 15:00 UTC

Changed in ossa:
status: In Progress → Fix Committed
Revision history for this message
John Garbutt (johngarbutt) wrote :

+2 on that patch, but looks like we are sorted anyways

Thierry Carrez (ttx)
Changed in nova:
milestone: none → icehouse-rc2
status: Confirmed → In Progress
Thierry Carrez (ttx)
Changed in nova:
importance: Undecided → High
information type: Private Security → Public Security
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (milestone-proposed)

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/86360

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote : Re: RBAC policy not enforced when adding a security group rule using EC2 API (CVE-2014-0167)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/86358
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d4056f8723cc6cefb28ff6e5a7c0df5ea77f82ef
Submitter: Jenkins
Branch: master

commit d4056f8723cc6cefb28ff6e5a7c0df5ea77f82ef
Author: Andrew Laski <email address hidden>
Date: Thu Mar 20 19:04:09 2014 -0400

    Add RBAC policy for ec2 API security groups calls

    The revoke_security_group_ingress, revoke_security_group_ingress, and
    delete_security_group calls in the ec2 API were not restricted by policy
    checks. This prevented a deployer from restricting their usage via
    roles or other checks. Checks have been added for these calls.

    Closes-Bug: #1290537
    Change-Id: I4bf681bedd68ed2216b429d34db735823e0a6189

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (milestone-proposed)

Reviewed: https://review.openstack.org/86360
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=87f57c0a2cc00a70edc87c5dc10bdefb6c01587b
Submitter: Jenkins
Branch: milestone-proposed

commit 87f57c0a2cc00a70edc87c5dc10bdefb6c01587b
Author: Andrew Laski <email address hidden>
Date: Thu Mar 20 19:04:09 2014 -0400

    Add RBAC policy for ec2 API security groups calls

    The revoke_security_group_ingress, revoke_security_group_ingress, and
    delete_security_group calls in the ec2 API were not restricted by policy
    checks. This prevented a deployer from restricting their usage via
    roles or other checks. Checks have been added for these calls.

    Closes-Bug: #1290537
    Change-Id: I4bf681bedd68ed2216b429d34db735823e0a6189
    (cherry picked from commit d4056f8723cc6cefb28ff6e5a7c0df5ea77f82ef)

Changed in nova:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/havana)

Reviewed: https://review.openstack.org/86361
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=dbb7dd03fea68120ef5ac9bbb1b3f184e3f2eacc
Submitter: Jenkins
Branch: stable/havana

commit dbb7dd03fea68120ef5ac9bbb1b3f184e3f2eacc
Author: Andrew Laski <email address hidden>
Date: Wed Apr 9 09:27:44 2014 -0400

    Add RBAC policy for ec2 API security groups calls

    The revoke_security_group_ingress, revoke_security_group_ingress, and
    delete_security_group calls in the ec2 API were not restricted by policy
    checks. This prevented a deployer from restricting their usage via
    roles or other checks. Checks have been added for these calls.

    Based on commit d4056f8723cc6cefb28ff6e5a7c0df5ea77f82ef but modified
    for the backport.

    Closes-Bug: #1290537
    Change-Id: I4bf681bedd68ed2216b429d34db735823e0a6189

summary: - RBAC policy not enforced when adding a security group rule using EC2 API
- (CVE-2014-0167)
+ [0SSA 2014-011] RBAC policy not enforced when adding a security group
+ rule using EC2 API (CVE-2014-0167)
Changed in ossa:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: icehouse-rc2 → 2014.1
Alan Pevec (apevec)
tags: removed: icehouse-rc-potential
To post a comment you must log in.
This report contains Public Security information  Edit
Everyone can see this security related information.

Other bug subscribers