Huge spikes in get_instance_nw_info since upgrading to grizzly

[Sam Morrison]

We have recently upgraded to grizzly and we have started seeing huge spikes on our network queue.

Our network queue sits on about 1 message per second usually but we get spikes of 200m/s every so often.
These spikes are all due to the message get_instance_nw_info.

We're pretty sure this is due to a user action and not some periodic task.
One thought is maybe something to do with security groups and source groups.
We have some tenants with 100+ instances all sharing the same security group source group.

We're using nova-network in FlatDHCP mode.

Happy to help debug this, just don't know what information is useful.

[Sam Morrison (sorrison)]

A little more information, I asked one of our big users who can trigger this at will on what they were doing and I'm now pretty sure this has to do with security groups.

They have approx. 50 instances which have a security group rule X to allow traffic from another security group rule Y.

When they launch an instance with security group Y it creates about 250 get_instance_nw_info messages on the network queue.

These guys regularly launch a large number of instances at the same time and therefore we get Nx250 messages.

We didn't see this in folsom.