Comment 0 for bug 969545

Revision history for this message
Dan Prince (dan-prince) wrote :

Nova does not currently enforce quotas on the number of security group rules an *authorized* user can create.

Using the EC2 API an authorized user could do the following to create 3000+ rules in the default security group:

for X in {1..1024}; do
euca-authorize -P icmp -p $X default
euca-authorize -P tcp -p $X default
euca-authorize -P udp -p $X default
done

** I choose 1024 as the upper limit of the privileged ports... you could go higher...

Each of these commands would translate into 3000+ iptables rules getting created per/instance. Furthermore... if you do this after you have instances created it causes quite a bit of thrashing of iptables rules on the Nova compute nodes ... to the extent that instances seem to get stuck in BUILD state with only a few nodes... and the nova compute.log file size grows quite large as well (Gigs of data) due to the fact that the iptables rules are getting logged (via log channel 'info'):

After a couple of instances and runs of this script I'm already at well over 10 gigs of data in Nova's compute.log file.

------

The ability to create this many iptables rules is a performance concern from both a networking and compute service prospective. I'm not sure what the limit on iptables rules is but 3000+ rules per instance on each nova compute host is certainly going to be a drain. Additionally the extra overhead of maintain this many rules can significantly slow down Nova compute's performance to manage instances.

Couple of fixes we should look at doing:

-Add a quota to limit the number of groups and rules per group to something like 20 groups/20.

-Tone down some of the log level info logging in the firewall drivers.

Optionally there are some optimizations that we could probably look at to combine some of the individual iptables rules into ranges. Probably best not to do that within the scope of this ticket however.

---

Lastly... we can do the same thing via the OSAPI using 'nova secgroup-add-rule' although it should be noted that you'd have to slow down things a bit on the OSAPI side due to rate limiting of the POST requests... So the OSAPI would slow down an attack on this front but wouldn't prevent it.