[OSSA 2013-020] Denial of Service in Nova network source security groups (CVE-2013-4185)

[Vish Ishaya]

The code for retrieving the list of ips to make source security groups is very inefficient. It is possible to create a set of rules using source security groups that will lead to an explosion of get_network_info calls.

Example:

# create a security group
nova secgroup-create foo foo

# add source group rules referencing the same group
for i in {1..10}; do
  nova secgroup-add-group-rule foo foo tcp $i $i
done

# launch 10 instances
nova boot --flavor 1 --image precise --num-instances 10 --security-groups foo test

When each instance boots it sends out a notification to each of the other instances to update their iptables rules. Here is what happens when the 10th instance boots:

for num instances in group that is referenced in a source rule: 10
  for num rules referencing a source group: 10
    for num instances in the source group: 10
      call get_nw_info()

That means 1000 individual requests to get_nw_info for a single instance launch. In this case you have You can see how these numbers could get out of hand very quickly, for example launching 20 instances:
instance number(n) : num calls (n^2) * 10
1 : 10
2 : 20
3 : 90
...
18 : 3240
19 : 3610
20 : 4000
...
$ python -c "print sum(n * n * 10 for n in xrange(1, 21))"
28700

28,700 calls to get_network info by the time the launch completes. In reality this number of calls creates a DOS where nova-network can't respond in time and instance launches start to timeout.

There are multiple fixes needed:

a) constructing a list of all needed instances instead of making a separate request for each rule.
b) a single call to retrieve nw_info (or at least fixed ip info) for multiple instances at once.

[Thierry Carrez (ttx)]

Not totally convinced we should issue an OSSA about this one... It's a bit of a slippery slope since authenticated users have a lot of ways to consume resources . Only wildly assymetric attacks should trigger OSSAs, but that may well be the case here.

That only affects Nova when used with nova-network (not Quantum), right ?

[Vish Ishaya (vishvananda)]

ttx: I'm not sure if it needs an OSSA, although this case is particularly nasty. It isn't just that you are using resources, you can cause the system to become so overloaded that all nova-computes in the system cannot respond to messages. So it basically borks the entire cloud.

[Russell Bryant (russellb)]

It definitely sounds nasty. It seems OSSA worthy to me. I guess we could use some more opinions. Sometimes I feel overly conservative when considering DOS vulnerabilities.

If I understand this correctly, the number we're talking about is blocking rpc calls made to the network service. Starting 20 instances isn't too hard, and resulting in 28,700 rpc calls in this example is pretty darn bad.

[Jeremy Stanley (fungi)]

This reasoning also takes us back to the earlier discussion about authenticated vs. unauthenticated and whether we can infer what that means for all deployers. With today's free trials and otherwise minimally vetted users I'm leaning toward disregarding that axis in appraising the severity of a risk from the project perspective.

[Thierry Carrez (ttx)]

Agreed on OSSA need. Now do we have a fix ? :)

[Jeremy Stanley (fungi)]

Proposed impact description...

Title: Denial of Service in Nova network source security groups
Reporter: Vishvananda Ishaya (Nebula)
Products: Nova
Affects: All versions

Vishvananda Ishaya from Nebula reported a denial of service
vulnerability in Nova's handling of network source security group
policy updates. By performing a large number of server creation
operations, the proportion of updates increases quadratically and
may overwhelm nova-network such that it is no longer able to service
other requests in a timely fashion. Only setups relying on
nova-network are affected.

[Thierry Carrez (ttx)]

Impact ++

[Thierry Carrez (ttx)]

Vish: do you have a fix for that, or should we look for someone else to do one ?

[Vish Ishaya (vishvananda)]

I don't have a fix yet no, just the method by which to construct one outlined in the bug report.

[Jeremy Stanley (fungi)]

Beating the drum on this one to see if we're any closer to having a fix, or whether we should pull in some additional development resources as subscribers... thoughts?

[Vish Ishaya (vishvananda)]

Found a much simpler solution to this after some experimenting. The original solution touches a ton of files but this one is pretty simple. It uses the cached ip addresses to construct the security group rules.

[Russell Bryant (russellb)]

nice work! +2 My only other question is whether any tests are needed to cover this.

[Vish Ishaya (vishvananda)]

Applies to grizzly with some very minor line continuation changes.

[Vish Ishaya (vishvananda)]

Folsom version

[Russell Bryant (russellb)]

+2 on the backports

[Jeremy Stanley (fungi)]

Note that we're going to be unable to land anything on stable/folsom until someone picks up nova bug 1200336 and devises a fix (also we'll need to manually merge https://review.openstack.org/36698 to d-g since we're deadlocked on multiple regressions in the folsom gate now).

[Russell Bryant (russellb)]

Just came across https://bugs.launchpad.net/nova/+bug/1202893 which looks like a duplicate ...

[Jeremy Stanley (fungi)]

Agreed, it looks like the same set of symptoms. I think we should open this bug public and mark that one as a duplicate, then use the normal review process to wrap things up from this point forward. Anyone disagree?

[Thierry Carrez (ttx)]

Nope, sounds good to me.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/39541

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/39543

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/folsom
Review: https://review.openstack.org/39544

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/39541
Committed: http://github.com/openstack/nova/commit/85aac04704350566d6b06aa7a3b99649946c672c
Submitter: Jenkins
Branch: master

commit 85aac04704350566d6b06aa7a3b99649946c672c
Author: Vishvananda Ishaya <email address hidden>
Date: Fri Jul 19 10:23:59 2013 -0700

    Use cached nwinfo for secgroup rules

    This stops a potential DOS with source security groups by using the
    db cached version of the network info instead of calling out to
    the network api multiple times.

    Fixes bug 1184041

    Change-Id: Id5f24ecf0e8cce60c27a9aecbc6e606c4c44d6b6

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/39543
Committed: http://github.com/openstack/nova/commit/52ad911963da4095b213952dee3a430fe0c4c30f
Submitter: Jenkins
Branch: stable/grizzly

commit 52ad911963da4095b213952dee3a430fe0c4c30f
Author: Vishvananda Ishaya <email address hidden>
Date: Fri Jul 19 10:23:59 2013 -0700

    Use cached nwinfo for secgroup rules

    This stops a potential DOS with source security groups by using the
    db cached version of the network info instead of calling out to
    the network api multiple times.

    Fixes bug 1184041

    Change-Id: Id5f24ecf0e8cce60c27a9aecbc6e606c4c44d6b6
    (cherry picked from commit 85aac04704350566d6b06aa7a3b99649946c672c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/39544
Committed: http://github.com/openstack/nova/commit/d4ee081c5c0a5132781235177c430ebcf72b0b0b
Submitter: Jenkins
Branch: stable/folsom

commit d4ee081c5c0a5132781235177c430ebcf72b0b0b
Author: Vishvananda Ishaya <email address hidden>
Date: Fri Jul 19 10:23:59 2013 -0700

    Use cached nwinfo for secgroup rules

    This stops a potential DOS with source security groups by using the
    db cached version of the network info instead of calling out to
    the network api multiple times.

    Fixes bug 1184041

    Change-Id: Id5f24ecf0e8cce60c27a9aecbc6e606c4c44d6b6
    (cherry picked from commit 85aac04704350566d6b06aa7a3b99649946c672c)

[Sam Morrison (sorrison)]

This has a side affect of breaking security group linking via source groups.

Example:

Secgroup "Server" has rule

+-------------+-----------+---------+----------+---------------+
| IP Protocol | From Port | To Port | IP Range | Source Group |
+-------------+-----------+---------+----------+---------------+
| tcp | 8140 | 8140 | | client |
+-------------+-----------+---------+----------+---------------+

Instance A is running and is associated with the "Server" secgroup

Now if a create a new instance with the "client" security group it should add a rule to instance A to all the new instance to talk to it.

This no longer gets added as it uses the cached version.

Restarting nova-compute will fix this and the rule will be applied.

[Thierry Carrez (ttx)]

@Sam: could you file a separate bug about this regression ? It will be easier to track rather than reusing the original bug.

[Sam Morrison (sorrison)]

Created new bug #1216720