[0SSA 2014-011] RBAC policy not enforced when adding a security group rule using EC2 API (CVE-2014-0167)

[Marc Heckmann]

It seems that when using the EC2 API, the security group implementation does not enforce RBAC policy for the add_rules, remove_rules, destroy and other functions (in compute/api.py). Only the add_to_instance and remove_from_instance functions enforce RBAC. This seems like an oversight for obvious reasons.

The Nova API security group implementation does enforce RBAC on these functions.

In addition, the add_to_instance and remove_from _instance functions which are wrapped in RBAC verification use the "compute:security_groups" action which is not even listed in the default /etc/nova/policy.json. The latter is confusing to users.

This is the case on Grizlly and at first glance, it doesn't look like this has changed in Havana.

[Marc Heckmann (marc-w-heckmann)]

What can I do to make this report more "complete"?

Thanks.

[Jeremy Stanley (fungi)]

I've marked the OpenStack Security Advisory task for this bug incomplete because the vulnerability management team is waiting for Nova core security reviewers to triage/confirm/reproduce the issue and determine a probable fix. Once we get closer to having a good grasp of the details there, the OSSA task will begin to separately progress through impact description drafting and review, CVE request and assignment, downstream notification and finally public disclosure (hopefully) synchronized with publication of the finalized patches. Hopefully that answers your questions.

[Marc Heckmann (marc-w-heckmann)]

Yes that answers my question. Thanks.

Let me know if there's anything I can do in the meantime.

[Andrew Laski (alaski)]

I've confirmed the behaviour, and it seems that it's likely been this way for a while.

The trend in the OpenStack API has been to move policy out of the compute api and into the API itself. Even the existing checks on add_to_instance/remove_from_instance are likely to go away at some point. So to keep in line with that the fix for this would be to add policy checks into the ec2 API.

The code for policy enforcement is non invasive and easily backportable but we'll want to set a default policy that won't cause a problem for deployments. As mentioned by Marc, the current "compute:security_groups" rule isn't listed which means it uses the default rule. The default for the default rule is to limit calls to admins or users who share a project_id with the security group. If we set up new policies for the security group calls that use the default rule it may break a use case that relies on the unrestricted behaviour currently available, but I believe that's ok. And using the default rule will put the new policies in line with the existing policies for security groups calls.

[Thierry Carrez (ttx)]

Trying to make sure this is an exploitable vulnerability...

if add_to_instance and remove_from_instance are protected by RBAC, can you actually do any harm using the unprotected functions ? What would be the attack scenario ? Would for example using remove_rules remove securitygroup rules from instances ?

[Marc Heckmann (marc-w-heckmann)]

Assuming the default case of deny all in a security group, the attack scenario would be to use add_rules to add rules to an instance.

Now, the problem with this bug is that users may get into a situation where there is a false sense of security.

For example, as a private-cloud operator, I want to restrict all Nova actions to users with the "Member" role on the tenant. Once that is done, I want to create a new role that allows only for read-only access to security group and instance information for auditing purposes. I can achieve this by editing the rule for the "compute_extension:security_groups" action in /etc/nova/policy.json. This will do what I want for the Nova API. The problem is that the EC2 API is left vulnerable and a user that does not have the Member role but only the read-only role can now add or delete rules from a security group.

I hope that helps to clarify things. Of course, this bug only affects users who are trying to create tighter access controls. The default configuration is not affected by this bug since it grants any role on a project/tenant full access.

[Thierry Carrez (ttx)]

OK, that makes sense, thanks for the explanation! We'll issue an advisory for this one.
Andrew: would you be up for a patch ?

[Andrew Laski (alaski)]

Thierry: yes, I can put a patch together for this.

[Andrew Laski (alaski)]

Here is an initial patch for review.

[Thierry Carrez (ttx)]

nova-coresec: care to review that one ?

[Thierry Carrez (ttx)]

A couple questions while I try to draft an impact description for the CVE:

* Mark, should we credit the company you work for, in addition to your name ? If yes, which company is it ?

* Any idea which versions of OpenStack should be considered vulnerable to this ? I suspect that would be when RBAC was added to openStack API, but not sure how to be more precise than that

[Marc Heckmann (marc-w-heckmann)]

I will check with my company and get back to you about the divulging the name.

For the versions, it sounds like this might go back pretty far. I can confirm that it affects Grizzly.

[Marc Heckmann (marc-w-heckmann)]

Hi, you can use the company name: Ubisoft

[Tristan Cacqueray (tristan-cacqueray)]

Impact description draft #1:

Title: RBAC policy not enforced in Nova EC2 API
Reporter: Marc Heckmann (Ubisoft)
Products: Nova
Versions: 2013.1 versions up to 2013.2.2

Description:
Marc Heckmann from Ubisoft reported a vulnerability in the Nova EC2 API security group implementation. RBAC policy are not enforced when using the EC2 API, in particular the add_rules, remove_rules and destroy methods. A restricted user may overcome his limitation by using EC2 API resulting in unauthorized action on security groups. Only setups using tighter access controls for Nova API are affected.

[Thierry Carrez (ttx)]

Suggestions:

Title: "RBAC policy not properly enforced in Nova EC2 API"

RBAC policy are not enforced -> "policies"

and replace last sentence: "Only setups using non-default RBAC rules for Nova may be affected".

[Tristan Cacqueray (tristan-cacqueray)]

@ttx Thanks!

Impact description draft #2:

Title: RBAC policy not properly enforced in Nova EC2 API
Reporter: Marc Heckmann (Ubisoft)
Products: Nova
Versions: 2013.1 versions up to 2013.2.2

Description:
Marc Heckmann from Ubisoft reported a vulnerability in the Nova EC2 API security group implementation. RBAC policies are not enforced when using the EC2 API, in particular the add_rules, remove_rules and destroy methods. A restricted user may overcome his limitation by using EC2 API resulting in unauthorized action on security groups. Only setups using non-default RBAC rules for Nova may be affected.

[Thierry Carrez (ttx)]

+1 on Impact description draft #2

[Russell Bryant (russellb)]

+2 on the patch.

I'm going to add cyeoh to this bug, as I think he'd be a good nova-core member to help review this fix.

[Jeremy Stanley (fungi)]

Tristan's impact description in comment #16 looks good to me.

[Christopher Yeoh (cyeoh-0)]

+1 on the impact description

Re: the patch - Although I agree it fixes the problem I'm not sure its the way we should do it.

We already have a policy "compute_extension:security_groups" which is used by the Nova API to enforce
access in a general way to all of these features. So I'm thinking that rather than introducing new policies
more specific such as compute:security_groups:remove_rules settings to fix the bug we should just use
the existing generic compute_extension:security_groups rule in the ec2 API.

Now I think there may be a reasonable argument for having per method type policy settings in the API
rather than general ones and we have generally been moving this way anyway, but if we do so we need to
be consistent across the Nova and EC2 APIs. So if a compute:security_groups:remove_rules policy is used
by the EC2 code it should also be checked by the Nova API code too. Otherwise deployers could be mislead into
thinking the policy is affecting the Nova API when it isn't.

I think unless there is a clear need now to have detailed policy checks I think for an immediate bug fix we
should just use the existing general security groups policy (which I think also means no policy file changes
are needed by deployers just the code update). And then if we want to have more fine grained policy checks
we can do this in Juno. And this needs to be done fairly carefully for the Nova API side to ensure minimal
upgrade impact.

And more generally we definitely need to ensure all possibly policy checks are in the policy file rather than
rely on defaults as deployers should not be expected to grep the code to see what switches are available :-) I will
submit a blueprint for that work in Juno.

[Tristan Cacqueray (tristan-cacqueray)]

@Andrew Laski, Could you please attach another patch for Havana ?
The current one in comment #9 does apply to stable/havana but tests are failing, I think we are missing some import:

NameError: global name 'extensions' is not defined in nova/api/ec2/cloud.py
NameError: global name 'common_policy' is not defined in nova/tests/

[Tristan Cacqueray (tristan-cacqueray)]

But then extensions module is lacking check_compute_policy method.

[John Garbutt (johngarbutt)]

Yeah, I am -1 for what chris said.

Seems like we should use the same policy as the core openstack API, otherwise people will have to configure new policy for an api they may not release that had turned on, or something like that.

I agree that we should have a more granular policy on both in the long term, but thats not really the fix for this bug, as I see it.

[Marc Heckmann (marc-w-heckmann)]

I was hesitant to say anything about the patch, but Christopher is right, keeping the previously hidden "compute:security_groups" policy doesn't make sense if people are already using "compute_extension:security_groups". We should just have one way to do this.

And yeah, as an operator, having more fine grained policy is a plus, but this could be done in a future release.

[Andrew Laski (alaski)]

Christopher: Works for me. I'm happy with a simpler solution for now if everyone agrees. I'll update shortly.
And Tristan, I'll work on a backport but I just wanted to get approval on the direction first.

I added the fine grained controls into the ec2 api because the OpenStack api has fine grained controls as well, they're just enforced at the compute api level. Actually I see now that I should have just added the @wrap_check_security_groups_policy decorator for the compute api methods that the ec2 api uses. But really the ec2 api and OpenStack api have different semantics, and make different calls, so I think it's fine for them to have different policies. That can be handled later in a non security review though.

[Andrew Laski (alaski)]

Patch for master

[Andrew Laski (alaski)]

stable/havana

[Tristan Cacqueray (tristan-cacqueray)]

Thank you Andrew, patches are passing tests on Havana and master (beside a seemingly unrelated test: nova.tests.virt.libvirt.test_libvirt.LibvirtConnTestCase.test_get_domain_info_with_more_return)

If core could approve it today I'll send the pre-OSSA with a disclosure date at:
2014-04-09 15:00 UTC

Thank you in advance!

[Christopher Yeoh (cyeoh-0)]

I'm +2 on the latest set of havana/master patches

[Tristan Cacqueray (tristan-cacqueray)]

the pre-OSSA have been sent.

Proposed public disclosure date/time:
2014-04-09 15:00 UTC

[John Garbutt (johngarbutt)]

+2 on that patch, but looks like we are sorted anyways

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/86360

[Tristan Cacqueray (tristan-cacqueray)]

Master review: https://review.openstack.org/#/c/86353/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/86358
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d4056f8723cc6cefb28ff6e5a7c0df5ea77f82ef
Submitter: Jenkins
Branch: master

commit d4056f8723cc6cefb28ff6e5a7c0df5ea77f82ef
Author: Andrew Laski <email address hidden>
Date: Thu Mar 20 19:04:09 2014 -0400

    Add RBAC policy for ec2 API security groups calls

    The revoke_security_group_ingress, revoke_security_group_ingress, and
    delete_security_group calls in the ec2 API were not restricted by policy
    checks. This prevented a deployer from restricting their usage via
    roles or other checks. Checks have been added for these calls.

    Closes-Bug: #1290537
    Change-Id: I4bf681bedd68ed2216b429d34db735823e0a6189

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/86360
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=87f57c0a2cc00a70edc87c5dc10bdefb6c01587b
Submitter: Jenkins
Branch: milestone-proposed

commit 87f57c0a2cc00a70edc87c5dc10bdefb6c01587b
Author: Andrew Laski <email address hidden>
Date: Thu Mar 20 19:04:09 2014 -0400

    Add RBAC policy for ec2 API security groups calls

    The revoke_security_group_ingress, revoke_security_group_ingress, and
    delete_security_group calls in the ec2 API were not restricted by policy
    checks. This prevented a deployer from restricting their usage via
    roles or other checks. Checks have been added for these calls.

    Closes-Bug: #1290537
    Change-Id: I4bf681bedd68ed2216b429d34db735823e0a6189
    (cherry picked from commit d4056f8723cc6cefb28ff6e5a7c0df5ea77f82ef)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/86361
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=dbb7dd03fea68120ef5ac9bbb1b3f184e3f2eacc
Submitter: Jenkins
Branch: stable/havana

commit dbb7dd03fea68120ef5ac9bbb1b3f184e3f2eacc
Author: Andrew Laski <email address hidden>
Date: Wed Apr 9 09:27:44 2014 -0400

    Add RBAC policy for ec2 API security groups calls

    The revoke_security_group_ingress, revoke_security_group_ingress, and
    delete_security_group calls in the ec2 API were not restricted by policy
    checks. This prevented a deployer from restricting their usage via
    roles or other checks. Checks have been added for these calls.

    Based on commit d4056f8723cc6cefb28ff6e5a7c0df5ea77f82ef but modified
    for the backport.

    Closes-Bug: #1290537
    Change-Id: I4bf681bedd68ed2216b429d34db735823e0a6189