cProfile - fix Security Groups hotfunctions
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
neutron |
Fix Released
|
High
|
Unassigned |
Bug Description
I used cProfile to profile neutron-ovs-agent (from neutron kilo 2015.1.0) as VMs are provisioned (see code sample below to reproduce).
I find a couple of functions in the IptablesManager scaling poorly with # of VMs (_modify_rules, and its callee find_last_entry).
As the # of current VMs doubles, the time spent in these functions to provision 10 new VMs also roughly doubles,
While we wait for the new IptablesTables firewall driver:
https:/
Can we improve the performance of the current iptables firewall on those 2 functions which do a lot of string processing on iptables rule strings checking for dups ?
Current: #VMs: 20, # iptables rules: 657,
provision 10 new VMs
ncalls tottime percall cumtime percall filename:
60 0.143 0.002 3.979 0.066 iptables_
25989 2.752 0.000 3.332 0.000 iptables_
Cumulative time spent in _find_last_entry: 3.3 sec
Current #VMs: 40, # iptables rules: 1277 ,
provision 10 new VMs
65 0.220 0.003 7.974 0.123 iptables_
38891 5.782 0.000 6.986 0.000 iptables_
Cumulative time spent in _find_last_entry: 6.9 sec
Current #VMs: 80, # iptables rules: 2517 ,
provision 10 new VMs
30 0.274 0.009 20.496 0.683 iptables_
43862 15.920 0.000 19.292 0.000 iptables_
Cumulative time spent in _find_last_entry: 19.2 sec
current #VMs: 160, # iptables rules: 4997,
provision 10 new VMs
20 0.375 0.019 49.255 2.463 iptables_
56478 39.275 0.001 47.629 0.001 iptables_
Cumulative time spent in _find_last_entry: 47.6 sec
-------
To Reproduce: THis is one way where we can control start/stop of profiling
based on presence of a file (/tmp/cprof) in the file-system
-------
Make following change to neutron_
import cProfile
import os.path
pr_enabled = False
pr = None
In OVSNeutronAgent add method:
def toggle_
global pr, pr_enabled
start = False
data = ""
fname = "vm.profile"
try:
if os.path.
except IOError as e:
if start and not pr_enabled:
pr = cProfile.Profile()
if not start and pr_enabled:
In polling loop:
-------------
This is another way to run the cProfile, but here, there is no way to control start/stop of profiling, and the profile includes the initialization also.
Run, neutron-ovs-agent as follows:
sudo -u neutron bash -c "/usr/bin/python -m cProfile -o /tmp/vm_1.profile /usr/bin/
description: | updated |
description: | updated |
description: | updated |
description: | updated |
description: | updated |
description: | updated |
description: | updated |
description: | updated |
description: | updated |
description: | updated |
description: | updated |
Assigning to Miguel for an initial look.