missing quotas on security group rules

Bug #969545 reported by Dan Prince
266
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Dan Prince
Essex
Fix Released
High
Unassigned
nova (Ubuntu)
Fix Released
Undecided
Unassigned
Precise
Fix Released
Undecided
Unassigned

Bug Description

Nova does not currently enforce quotas on the number of security group rules an *authorized* user can create.

Using the EC2 API an authorized user could do the following to create 2000+ rules in the default security group:

for X in {1..1024}; do
euca-authorize -P tcp -p $X default
euca-authorize -P udp -p $X default
done

** I choose 1024 as the upper limit of the privileged ports... you could go higher...

Each of these commands would translate into 2000+ iptables rules getting created per/instance. Furthermore... if you do this after you have instances created it causes quite a bit of thrashing of iptables rules on the Nova compute nodes ... to the extent that instances seem to get stuck in BUILD state with only a few nodes... and the nova compute.log file size grows quite large as well (Gigs of data) due to the fact that the iptables rules are getting logged (via log channel 'info'):

After a couple of instances and runs of this script I'm already at well over 10 gigs of data in Nova's compute.log file.

------

The ability to create this many iptables rules is a performance concern from both a networking and compute service prospective. I'm not sure what the limit on iptables rules is but 2000+ rules per instance on each nova compute host is certainly going to be a drain. Additionally the extra overhead of maintain this many rules can significantly slow down Nova compute's performance to manage instances.

Couple of fixes we should look at doing:

-Add a quota to limit the number of groups and rules per group to something like 20 groups/20.

-Tone down some of the log level info logging in the firewall drivers.

Optionally there are some optimizations that we could probably look at to combine some of the individual iptables rules into ranges. Probably best not to do that within the scope of this ticket however.

---

Lastly... we can do the same thing via the OSAPI using 'nova secgroup-add-rule' although it should be noted that you'd have to slow down things a bit on the OSAPI side due to rate limiting of the POST requests... So the OSAPI would slow down an attack on this front but wouldn't prevent it.

CVE References

Revision history for this message
Dan Prince (dan-prince) wrote :

The attached patch should tone down some of the excessive logging in the firewall driver.

Changed in nova:
status: New → In Progress
importance: Undecided → High
assignee: nobody → Dan Prince (dan-prince)
Revision history for this message
Russell Bryant (russellb) wrote :

Thanks, Dan!

I spoke to ttx about this a bit. We both think that this issue isn't severe enough to warrant holding up the Essex release and that we can roll this out as a security patch the week after Essex. What do you think?

Revision history for this message
Russell Bryant (russellb) wrote :

Adding nova PTL

Revision history for this message
Dan Prince (dan-prince) wrote :

Russel: sounds fine to me.

Dan Prince (dan-prince)
description: updated
Revision history for this message
Dan Prince (dan-prince) wrote :

Attached patch for upstream Folsom.

Revision history for this message
Russell Bryant (russellb) wrote :

VMT members: I'll coordinate this one.

I just spoke with dprince to sync up on what's next for this issue. We will need patches for master, stable/essex, and stable/diablo. Once all of those are ready, we need to get them pre-approved on this bug so that they can be expedited through gerrit when release time comes.

Revision history for this message
Dan Prince (dan-prince) wrote :

Attached security groups patch for Diablo.

Revision history for this message
Dan Prince (dan-prince) wrote :

Attach Essex security group quotas patch

Revision history for this message
Dan Prince (dan-prince) wrote :

Rebase and re-attach patch for upstream (Folsom)

Revision history for this message
Vish Ishaya (vishvananda) wrote :

nice work dan. Looks good to me

Revision history for this message
Russell Bryant (russellb) wrote :

Regarding the release date, we could do Thursday, April 19th, but I need a commitment from you guys to get the patch approved through gerrit while we're at the conference. If you're not comfortable with that, let's push it to Tuesday, April 24.

Proposed description:

Title: No quota enforced on security group rules
Impact: High
Reporter: Dan Prince <email address hidden>
Products: Nova
Affects: All versions

Description:
Dan Prince reported a vulnerability in Nova. He discovered that there was no limit on the number of security group rules a user can create. By creating a very large set of rules, an unreasonable number of iptables rules will be created on compute nodes, resulting in a denial of service.

Revision history for this message
Dan Prince (dan-prince) wrote :

I'm Okay with adding it at the conference. I'd like to avoid bit rot as much as possible.

Also, added Vek since he has done some quota work recently too.

Revision history for this message
Kevin L. Mitchell (klmitch) wrote :

in fact, I'm working on a pretty complete refactoring of quotas which will, among other things, make adding new quotas nearly trivial…but I doubt I'll have it finished before the summit.

Revision history for this message
Russell Bryant (russellb) wrote :

Ok, notification sent. We'll do Thursday, April 19th.

Revision history for this message
Mark McLoughlin (markmc) wrote :

It's good to see the diablo and essex patches don't require a schema change - we're not able to introduce new migrations on the stable branch.

All looks good to me. Nice work guys

Dan Prince (dan-prince)
visibility: private → public
Revision history for this message
Mark McLoughlin (markmc) wrote :
Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/6656
Committed: http://github.com/openstack/nova/commit/b22f17cd09c8fa0447503b7a5a84477d0a943d18
Submitter: Jenkins
Branch: master

commit b22f17cd09c8fa0447503b7a5a84477d0a943d18
Author: Dan Prince <email address hidden>
Date: Mon Apr 2 13:01:42 2012 -0400

    Logging updates in IptablesFirewallDriver.

    Update IptablesFirewallDriver so that it uses the debug log channel
    when logging specifics about each firewall rule.

    Fixes LP Bug #969545.

    Change-Id: Ie50a3607fff17f032d577af8a87d5f4582bcc919

Revision history for this message
Anne Gentle (annegentle) wrote :

I'd like these types of security fixes to be included in the docs.openstack.org documentation and indicated in the revision history table for the Compute Admin manual.

tags: added: docimpact
Revision history for this message
Thierry Carrez (ttx) wrote :

This one should still have an advisory published ?

Revision history for this message
Russell Bryant (russellb) wrote :

An advisory went out last Thursday: https://lists.launchpad.net/openstack/msg10268.html

Revision history for this message
Thierry Carrez (ttx) wrote :

OK, removing team subscription then.

Thierry Carrez (ttx)
Changed in nova:
milestone: none → folsom-1
status: Fix Committed → Fix Released
Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "logging patch" of this bug report has been identified as being a patch. The ubuntu-reviewers team has been subscribed to the bug report so that they can review the patch. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-reviewers team please also unsubscribe the team from this bug report.

[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]

tags: added: patch
Chuck Short (zulcss)
Changed in nova (Ubuntu):
status: New → Fix Released
Revision history for this message
Jamie Strandboge (jdstrand) wrote :

This was fixed in 12.04 several weeks ago in http://www.ubuntu.com/usn/usn-1438-1/

Changed in nova (Ubuntu Precise):
status: New → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: folsom-1 → 2012.2
Sean Dague (sdague)
no longer affects: nova/diablo
To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.