compute node's OVS agent takes long time to scan sync all port's stat and update port security rules

Bug #1194438 reported by Raid
60
This bug affects 12 people
Affects Status Importance Assigned to Milestone
neutron
High
Peter Feiner
Grizzly
High
Édouard Thuleau

Bug Description

Our ENV:
1. Around 80 VMs in a compute node (Dell620, 2x8core CPU, 192G RAM)
2. Using OVS and GRE tunnel
3. Overlapping IP enabled
4. Quantum port security enabled

Symptom:
1. 'alive stats' of compute node's OVS agent in 'quantum agent-list' is keeping change.
2. Can't get DHCP address and network is not connective for a period of time when new created VM boot up.

Possible cause
1. The looping tasks of OVS agent take a long time, e.g. retrive all ovs port information. 'quantum-rootwrap' is very low performance, it takes around 0.2s for scaning each port, so 80 ports for 80 vm will take around 16s, the default 'report_interval' is 4s and 'agent_down_time' is 5s. It will be 10 times faster if using 'sudo' as the 'root_helper'.
2. OVS agent is updating 'port filter for' all ports in the compute node along with new VM creation. Each process for one port will take around 1s, so 80 ports will take around 80s. If there are 3 VMs created at same time in the same compute node, OVS agent will update 'port filter for' all ports 3 times, which means it will take around 240s.
3. Seems OVS agent set appropriate 'tag' and 'flow' for a port of new created VM after the above process of 'update port filter' completed.

Log pieces:
2013-06-25 11:40:53 WARNING [quantum.openstack.common.loopingcall] task run outlasted interval by 26.307028 sec
2013-06-25 11:46:41 WARNING [quantum.openstack.common.loopingcall] task run outlasted interval by 114.839855 sec

2013-06-25 11:45:12 DEBUG [quantum.agent.linux.iptables_firewall] Updating device (5f22d381-add6-4f87-8eac-780fe4e43b55) filter
2013-06-25 11:45:13 DEBUG [quantum.agent.securitygroups_rpc] Update port filter for {u'status': u'ACTIVE', u'security_group_source_groups': [u'1347300d-e008-47e5-9b8e-57b4296e7357', u'1347300d-e008-47e5-9b8e-57b4296e7357'], u'name': u'', u'admin_state_up': True, u'network_id': u'846ff546-fb71-44e9-974e-68c1f0f5f260', u'tenant_id': u'8fecb11221ef42c6949ee7ce5c31f8b2', u'security_group_rules': [{u'ethertype': u'IPv6', u'direction': u'egress', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'egress', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.73/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.61/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.4.4/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.104/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.36/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.86/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.76/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.40/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.31/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.25/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.35/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.89/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.13/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.46/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.57/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.26/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'10.10.0.7/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.88/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.67/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.91/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'172.30.0.4/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.33/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.68/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.27/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.63/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.17/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'112.21.0.4/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.93/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.78/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.82/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'172.12.0.4/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.8/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'172.12.0.3/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.34/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.2/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'222.128.0.5/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.62/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.4/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.7/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.22/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.58/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.54/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.18/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'10.10.0.4/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.83/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'172.148.0.6/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.41/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.51/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.11/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.98/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.48/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.28/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.37/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.30/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.77/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.99/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.19/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.50/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.60/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.55/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.12/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.103/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.95/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'172.12.0.6/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.79/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.16/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.45/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.84/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.53/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.92/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.102/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.71/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.75/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.69/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.81/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.72/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.10/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.5/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.100/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.43/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.24/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.101/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.66/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.14/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.56/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.3/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.85/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.44/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.90/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'172.30.0.5/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.6/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.49/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.87/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.59/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.38/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.29/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.21/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.65/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'172.148.0.5/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.47/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.80/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.32/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'172.11.0.3/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.42/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.94/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.97/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.39/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.9/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.70/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.15/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.96/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.74/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'ethertype': u'IPv4', u'direction': u'ingress', u'remote_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357', u'source_ip_prefix': u'192.168.1.52/32', u'security_group_id': u'1347300d-e008-47e5-9b8e-57b4296e7357'}, {u'source_port_range_min': 67, u'direction': u'ingress', u'protocol': u'udp', u'ethertype': u'IPv4', u'port_range_max': 68, u'source_port_range_max': 67, u'port_range_min': 68, u'source_ip_prefix': u'10.10.0.2/32'}, {u'source_port_range_min': 67, u'direction': u'ingress', u'protocol': u'udp', u'ethertype': u'IPv4', u'port_range_max': 68, u'source_port_range_max': 67, u'port_range_min': 68, u'source_ip_prefix': u'112.21.0.2/32'}, {u'source_port_range_min': 67, u'direction': u'ingress', u'protocol': u'udp', u'ethertype': u'IPv4', u'port_range_max': 68, u'source_port_range_max': 67, u'port_range_min': 68, u'source_ip_prefix': u'172.11.0.2/32'}, {u'source_port_range_min': 67, u'direction': u'ingress', u'protocol': u'udp', u'ethertype': u'IPv4', u'port_range_max': 68, u'source_port_range_max': 67, u'port_range_min': 68, u'source_ip_prefix': u'172.12.0.2/32'}, {u'source_port_range_min': 67, u'direction': u'ingress', u'protocol': u'udp', u'ethertype': u'IPv4', u'port_range_max': 68, u'source_port_range_max': 67, u'port_range_min': 68, u'source_ip_prefix': u'172.148.0.2/32'}, {u'source_port_range_min': 67, u'direction': u'ingress', u'protocol': u'udp', u'ethertype': u'IPv4', u'port_range_max': 68, u'source_port_range_max': 67, u'port_range_min': 68, u'source_ip_prefix': u'172.16.0.2/32'}, {u'source_port_range_min': 67, u'direction': u'ingress', u'protocol': u'udp', u'ethertype': u'IPv4', u'port_range_max': 68, u'source_port_range_max': 67, u'port_range_min': 68, u'source_ip_prefix': u'172.30.0.2/32'}, {u'source_port_range_min': 67, u'direction': u'ingress', u'protocol': u'udp', u'ethertype': u'IPv4', u'port_range_max': 68, u'source_port_range_max': 67, u'port_range_min': 68, u'source_ip_prefix': u'192.168.0.2/32'}, {u'source_port_range_min': 67, u'direction': u'ingress', u'protocol': u'udp', u'ethertype': u'IPv4', u'port_range_max': 68, u'source_port_range_max': 67, u'port_range_min': 68, u'source_ip_prefix': u'192.168.1.20/32'}, {u'source_port_range_min': 67, u'direction': u'ingress', u'protocol': u'udp', u'ethertype': u'IPv4', u'port_range_max': 68, u'source_port_range_max': 67, u'port_range_min': 68, u'source_ip_prefix': u'192.168.4.2/32'}, {u'source_port_range_min': 67, u'direction': u'ingress', u'protocol': u'udp', u'ethertype': u'IPv4', u'port_range_max': 68, u'source_port_range_max': 67, u'port_range_min': 68, u'source_ip_prefix': u'192.21.0.2/32'}, {u'source_port_range_min': 67, u'direction': u'ingress', u'protocol': u'udp', u'ethertype': u'IPv4', u'port_range_max': 68, u'source_port_range_max': 67, u'port_range_min': 68, u'source_ip_prefix': u'222.128.0.2/32'}, {u'source_port_range_min': 67, u'direction': u'ingress', u'protocol': u'udp', u'ethertype': u'IPv4', u'port_range_max': 68, u'source_port_range_max': 67, u'port_range_min': 68, u'source_ip_prefix': u'10.10.0.3/32'}, {u'source_port_range_min': 67, u'direction': u'ingress', u'protocol': u'udp', u'ethertype': u'IPv4', u'port_range_max': 68, u'source_port_range_max': 67, u'port_range_min': 68, u'source_ip_prefix': u'112.21.0.3/32'}, {u'source_port_range_min': 67, u'direction': u'ingress', u'protocol': u'udp', u'ethertype': u'IPv4', u'port_range_max': 68, u'source_port_range_max': 67, u'port_range_min': 68, u'source_ip_prefix': u'172.11.0.4/32'}, {u'source_port_range_min': 67, u'direction': u'ingress', u'protocol': u'udp', u'ethertype': u'IPv4', u'port_range_max': 68, u'source_port_range_max': 67, u'port_range_min': 68, u'source_ip_prefix': u'172.12.0.5/32'}, {u'source_port_range_min': 67, u'direction': u'ingress', u'protocol': u'udp', u'ethertype': u'IPv4', u'port_range_max': 68, u'source_port_range_max': 67, u'port_range_min': 68, u'source_ip_prefix': u'172.148.0.3/32'}, {u'source_port_range_min': 67, u'direction': u'ingress', u'protocol': u'udp', u'ethertype': u'IPv4', u'port_range_max': 68, u'source_port_range_max': 67, u'port_range_min': 68, u'source_ip_prefix': u'172.16.0.3/32'}, {u'source_port_range_min': 67, u'direction': u'ingress', u'protocol': u'udp', u'ethertype': u'IPv4', u'port_range_max': 68, u'source_port_range_max': 67, u'port_range_min': 68, u'source_ip_prefix': u'172.30.0.3/32'}, {u'source_port_range_min': 67, u'direction': u'ingress', u'protocol': u'udp', u'ethertype': u'IPv4', u'port_range_max': 68, u'source_port_range_max': 67, u'port_range_min': 68, u'source_ip_prefix': u'192.168.0.3/32'}, {u'source_port_range_min': 67, u'direction': u'ingress', u'protocol': u'udp', u'ethertype': u'IPv4', u'port_range_max': 68, u'source_port_range_max': 67, u'port_range_min': 68, u'source_ip_prefix': u'192.168.1.23/32'}, {u'source_port_range_min': 67, u'direction': u'ingress', u'protocol': u'udp', u'ethertype': u'IPv4', u'port_range_max': 68, u'source_port_range_max': 67, u'port_range_min': 68, u'source_ip_prefix': u'192.168.4.3/32'}, {u'source_port_range_min': 67, u'direction': u'ingress', u'protocol': u'udp', u'ethertype': u'IPv4', u'port_range_max': 68, u'source_port_range_max': 67, u'port_range_min': 68, u'source_ip_prefix': u'192.21.0.3/32'}, {u'source_port_range_min': 67, u'direction': u'ingress', u'protocol': u'udp', u'ethertype': u'IPv4', u'port_range_max': 68, u'source_port_range_max': 67, u'port_range_min': 68, u'source_ip_prefix': u'222.128.0.3/32'}], u'device_owner': u'compute:None', u'mac_address': u'fa:16:3e:4a:7d:ac', u'device': u'395f4003-83e8-4c12-87e9-e10b4bdf2a73', u'fixed_ips': [u'192.168.1.64'], u'id': u'395f4003-83e8-4c12-87e9-e10b4bdf2a73', u'security_groups': [u'1347300d-e008-47e5-9b8e-57b4296e7357'], u'device_id': u'640abea7-4583-4da0-911f-a994b049160c'}

Tags: ovs Edit Tag help
Revision history for this message
Raid (raid-chan) wrote :
Download full text (8.7 KiB)

Further investigation:

looks like the default 'ingress' rule of quantum security group is using parameter ‘--remote_group_id’ and point to it self, the description of this is ‘Selecting a security group as the source will allow any other instance in that security group access to any other instance via this rule.‘.

In this case, each port in every created instance which is assigned same security group will be scanned and refreshed new 'firewall rule' set when new port created along with the creation of VM.

So in one scenario, in a large scale of cloud, there are hundred of VMs are assigned same security rule, then the nightmare happended. Each time to create a new VM, hundred of ports of rest created VMs will be scanned and updated with hundred of iptable-rules (most of rules is allowing access from the IP of existed ports ), the whole process will take very long time to complete. Further more, the network of new created VM will not be able to connect until the previous process completed, because OVS agent set 'tag' and 'flow' for the new created port after the existed ports been updated.

Security Group:

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| description | default |
| id | 64b7a259-7f6b-4fa0-882d-c88ace616466 |
| name | default ...

Read more...

Revision history for this message
Peter Feiner (pete5) wrote :

I had the same performance problems. I have local patches that solve them. I will take ownership of this bug and send my patches for review.

Changed in neutron:
assignee: nobody → Peter Feiner (pete5)
Revision history for this message
Peter Feiner (pete5) wrote :

I'm not sure why gerrit didn't automatically link the review to this bug report. Anyhow, here's the first review: https://review.openstack.org/#/c/34812/. I'll submit the second patch for review tomorrow.

Changed in neutron:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to quantum (master)

Fix proposed to branch: master
Review: https://review.openstack.org/34872

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to quantum (master)

Reviewed: https://review.openstack.org/34812
Committed: http://github.com/openstack/quantum/commit/364e97b44158976dca923bcf9ff5dff926da5dc4
Submitter: Jenkins
Branch: master

commit 364e97b44158976dca923bcf9ff5dff926da5dc4
Author: Peter Feiner <email address hidden>
Date: Tue Jun 18 20:48:30 2013 +0000

    faster quantum-openvswitch-agent periodic reports

    One of two patches that fixes bug #1194438.

    Instead of spawning ovs-vsctl for every port on br-int, we just dump the
    required columns from the Interfaces table and grab the rows that we need. This
    is a big win because the time it takes for ovs-vsctl to connect to the
    openvswitch controller is orders of magnitude greater than the time it takes to
    parse the rows. In practice, instead of taking roughly 1s per interface, the
    agent's periodic task only takes 1s in total.

    Change-Id: Idbf32c38e0c4a9c9634c1e4f0e79bd556b720493

Changed in neutron:
status: In Progress → Fix Committed
tags: added: grizzly-backport-potential
tags: added: ovs
Changed in neutron:
milestone: none → havana-2
importance: Undecided → Critical
importance: Critical → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to quantum (stable/grizzly)

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/35512

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to quantum (stable/grizzly)

Reviewed: https://review.openstack.org/35512
Committed: http://github.com/openstack/quantum/commit/5f326710f392d71f33820e91e9127627f7a8f4b3
Submitter: Jenkins
Branch: stable/grizzly

commit 5f326710f392d71f33820e91e9127627f7a8f4b3
Author: Peter Feiner <email address hidden>
Date: Tue Jun 18 20:48:30 2013 +0000

    faster quantum-openvswitch-agent periodic reports

    One of two patches that fixes bug #1194438.

    Instead of spawning ovs-vsctl for every port on br-int, we just dump the
    required columns from the Interfaces table and grab the rows that we need. This
    is a big win because the time it takes for ovs-vsctl to connect to the
    openvswitch controller is orders of magnitude greater than the time it takes to
    parse the rows. In practice, instead of taking roughly 1s per interface, the
    agent's periodic task only takes 1s in total.

    Change-Id: Idbf32c38e0c4a9c9634c1e4f0e79bd556b720493
    (cherry picked from commit 364e97b44158976dca923bcf9ff5dff926da5dc4)

tags: added: in-stable-grizzly
Revision history for this message
Peter Feiner (pete5) wrote :

Changed back to in-progress because the second patch is still being reviewed: https://review.openstack.org/#/c/34872/

Changed in neutron:
status: Fix Committed → In Progress
Revision history for this message
Alan Pevec (apevec) wrote :

Édouard, could you also take care of backporting the 2nd Peter's patch when it's merged on master?

tags: removed: in-stable-grizzly
Changed in neutron:
milestone: havana-2 → havana-3
Revision history for this message
Édouard Thuleau (ethuleau) wrote :

Alan: yes, I can. I watch his review to be aware.

Alan Pevec (apevec)
Changed in neutron:
status: In Progress → Fix Committed
Revision history for this message
Alan Pevec (apevec) wrote :

In the meantime, https://review.openstack.org/34872 was merged to master and we entered stable/grizzly freeze for 2013.1.3 without backport to stable/grizzly.
Since this is marked as "High" I'll go ahead and propose it as a freeze exception.

Revision history for this message
Alan Pevec (apevec) wrote :
Thierry Carrez (ttx)
Changed in neutron:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in neutron:
milestone: havana-3 → 2013.2
Alan Pevec (apevec)
tags: removed: grizzly-backport-potential
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers