| 2015-09-04 20:45:55 |
Ramu Ramamurthy |
bug |
|
|
added bug |
| 2015-09-04 20:46:59 |
Ramu Ramamurthy |
description |
I used cProfile to profile neutron-ovs-agent (from neutron 2015.1.0) as VMs are provisioned (see code sample below to reproduce).
I find a couple of functions in the IptablesManager scaling poorly with # of VMs (_modify_rules, and its callee find_last_entry).
As the # of current VMs doubles, the time spent in these functions to provision 10 new VMs also roughly doubles,
While we wait for the new IptablesTables firewall driver:
https://blueprints.launchpad.net/neutron/+spec/new-iptables-driver
Can we improve the performance of the current iptables firewall driver to perform better ?
Current: #VMs: 20, # SG rules: 657,
provision 10 new VMs
60 0.143 0.002 3.979 0.066 iptables_manager.py:511(_modify_rules)
25989 2.752 0.000 3.332 0.000 iptables_manager.py:504(_find_last_entry)
Current #VMs: 40, # SG rules: 1277 ,
provision 10 new VMs
65 0.220 0.003 7.974 0.123 iptables_manager.py:511(_modify_rules)
38891 5.782 0.000 6.986 0.000 iptables_manager.py:504(_find_last_entry)
Current #VMs: 80, # SG rules: 2517 ,
provision 10 new VMs
30 0.274 0.009 20.496 0.683 iptables_manager.py:511(_modify_rules)
43862 15.920 0.000 19.292 0.000 iptables_manager.py:504(_find_last_entry)
current #VMs: 160, #SG rules: 4997,
provision 10 new VMs
20 0.375 0.019 49.255 2.463 iptables_manager.py:511(_modify_rules)
56478 39.275 0.001 47.629 0.001 iptables_manager.py:504(_find_last_entry)
To Reproduce:
-------------------
Make following change to neutron_ovs_agent.py to enable/disable cProfile for a given scenario.
< import cProfile
< import os.path
< pr_enabled = False
< pr = None
In OVSNeutronAgent add method:
< def toggle_cprofile(self):
< global pr, pr_enabled
< start = False
< data = ""
< fname = "vm.profile"
< try:
< if os.path.isfile("/tmp/cprof"):
< start = True
< except IOError as e:
< LOG.warn("Error %s", e.strerror)
<
< if start and not pr_enabled:
< pr = cProfile.Profile()
< pr.enable()
< pr_enabled = True
< LOG.warn("enabled cprofile")
< if not start and pr_enabled:
< pr.disable()
< pr.create_stats()
< pr.dump_stats("/tmp/%s"%fname)
< pr_enabled = False
< LOG.warn("disabled cprofile")
In polling loop:
< self.toggle_cprofile() |
I used cProfile to profile neutron-ovs-agent (from neutron 2015.1.0) as VMs are provisioned (see code sample below to reproduce).
I find a couple of functions in the IptablesManager scaling poorly with # of VMs (_modify_rules, and its callee find_last_entry).
As the # of current VMs doubles, the time spent in these functions to provision 10 new VMs also roughly doubles,
While we wait for the new IptablesTables firewall driver:
https://blueprints.launchpad.net/neutron/+spec/new-iptables-driver
Can we improve the performance of the current iptables firewall on those 2 functions which do a lot of string processing ?
Current: #VMs: 20, # SG rules: 657,
provision 10 new VMs
60 0.143 0.002 3.979 0.066 iptables_manager.py:511(_modify_rules)
25989 2.752 0.000 3.332 0.000 iptables_manager.py:504(_find_last_entry)
Current #VMs: 40, # SG rules: 1277 ,
provision 10 new VMs
65 0.220 0.003 7.974 0.123 iptables_manager.py:511(_modify_rules)
38891 5.782 0.000 6.986 0.000 iptables_manager.py:504(_find_last_entry)
Current #VMs: 80, # SG rules: 2517 ,
provision 10 new VMs
30 0.274 0.009 20.496 0.683 iptables_manager.py:511(_modify_rules)
43862 15.920 0.000 19.292 0.000 iptables_manager.py:504(_find_last_entry)
current #VMs: 160, #SG rules: 4997,
provision 10 new VMs
20 0.375 0.019 49.255 2.463 iptables_manager.py:511(_modify_rules)
56478 39.275 0.001 47.629 0.001 iptables_manager.py:504(_find_last_entry)
To Reproduce:
-------------------
Make following change to neutron_ovs_agent.py to enable/disable cProfile for a given scenario.
< import cProfile
< import os.path
< pr_enabled = False
< pr = None
In OVSNeutronAgent add method:
< def toggle_cprofile(self):
< global pr, pr_enabled
< start = False
< data = ""
< fname = "vm.profile"
< try:
< if os.path.isfile("/tmp/cprof"):
< start = True
< except IOError as e:
< LOG.warn("Error %s", e.strerror)
<
< if start and not pr_enabled:
< pr = cProfile.Profile()
< pr.enable()
< pr_enabled = True
< LOG.warn("enabled cprofile")
< if not start and pr_enabled:
< pr.disable()
< pr.create_stats()
< pr.dump_stats("/tmp/%s"%fname)
< pr_enabled = False
< LOG.warn("disabled cprofile")
In polling loop:
< self.toggle_cprofile() |
|
| 2015-09-04 20:54:08 |
Ramu Ramamurthy |
description |
I used cProfile to profile neutron-ovs-agent (from neutron 2015.1.0) as VMs are provisioned (see code sample below to reproduce).
I find a couple of functions in the IptablesManager scaling poorly with # of VMs (_modify_rules, and its callee find_last_entry).
As the # of current VMs doubles, the time spent in these functions to provision 10 new VMs also roughly doubles,
While we wait for the new IptablesTables firewall driver:
https://blueprints.launchpad.net/neutron/+spec/new-iptables-driver
Can we improve the performance of the current iptables firewall on those 2 functions which do a lot of string processing ?
Current: #VMs: 20, # SG rules: 657,
provision 10 new VMs
60 0.143 0.002 3.979 0.066 iptables_manager.py:511(_modify_rules)
25989 2.752 0.000 3.332 0.000 iptables_manager.py:504(_find_last_entry)
Current #VMs: 40, # SG rules: 1277 ,
provision 10 new VMs
65 0.220 0.003 7.974 0.123 iptables_manager.py:511(_modify_rules)
38891 5.782 0.000 6.986 0.000 iptables_manager.py:504(_find_last_entry)
Current #VMs: 80, # SG rules: 2517 ,
provision 10 new VMs
30 0.274 0.009 20.496 0.683 iptables_manager.py:511(_modify_rules)
43862 15.920 0.000 19.292 0.000 iptables_manager.py:504(_find_last_entry)
current #VMs: 160, #SG rules: 4997,
provision 10 new VMs
20 0.375 0.019 49.255 2.463 iptables_manager.py:511(_modify_rules)
56478 39.275 0.001 47.629 0.001 iptables_manager.py:504(_find_last_entry)
To Reproduce:
-------------------
Make following change to neutron_ovs_agent.py to enable/disable cProfile for a given scenario.
< import cProfile
< import os.path
< pr_enabled = False
< pr = None
In OVSNeutronAgent add method:
< def toggle_cprofile(self):
< global pr, pr_enabled
< start = False
< data = ""
< fname = "vm.profile"
< try:
< if os.path.isfile("/tmp/cprof"):
< start = True
< except IOError as e:
< LOG.warn("Error %s", e.strerror)
<
< if start and not pr_enabled:
< pr = cProfile.Profile()
< pr.enable()
< pr_enabled = True
< LOG.warn("enabled cprofile")
< if not start and pr_enabled:
< pr.disable()
< pr.create_stats()
< pr.dump_stats("/tmp/%s"%fname)
< pr_enabled = False
< LOG.warn("disabled cprofile")
In polling loop:
< self.toggle_cprofile() |
I used cProfile to profile neutron-ovs-agent (from neutron kilo 2015.1.0) as VMs are provisioned (see code sample below to reproduce).
I find a couple of functions in the IptablesManager scaling poorly with # of VMs (_modify_rules, and its callee find_last_entry).
As the # of current VMs doubles, the time spent in these functions to provision 10 new VMs also roughly doubles,
While we wait for the new IptablesTables firewall driver:
https://blueprints.launchpad.net/neutron/+spec/new-iptables-driver
Can we improve the performance of the current iptables firewall on those 2 functions which do a lot of string processing ?
Current: #VMs: 20, # SG rules: 657,
provision 10 new VMs
60 0.143 0.002 3.979 0.066 iptables_manager.py:511(_modify_rules)
25989 2.752 0.000 3.332 0.000 iptables_manager.py:504(_find_last_entry)
Current #VMs: 40, # SG rules: 1277 ,
provision 10 new VMs
65 0.220 0.003 7.974 0.123 iptables_manager.py:511(_modify_rules)
38891 5.782 0.000 6.986 0.000 iptables_manager.py:504(_find_last_entry)
Current #VMs: 80, # SG rules: 2517 ,
provision 10 new VMs
30 0.274 0.009 20.496 0.683 iptables_manager.py:511(_modify_rules)
43862 15.920 0.000 19.292 0.000 iptables_manager.py:504(_find_last_entry)
current #VMs: 160, #SG rules: 4997,
provision 10 new VMs
20 0.375 0.019 49.255 2.463 iptables_manager.py:511(_modify_rules)
56478 39.275 0.001 47.629 0.001 iptables_manager.py:504(_find_last_entry)
To Reproduce:
-------------------
Make following change to neutron_ovs_agent.py to enable/disable cProfile for a given scenario.
< import cProfile
< import os.path
< pr_enabled = False
< pr = None
In OVSNeutronAgent add method:
< def toggle_cprofile(self):
< global pr, pr_enabled
< start = False
< data = ""
< fname = "vm.profile"
< try:
< if os.path.isfile("/tmp/cprof"):
< start = True
< except IOError as e:
< LOG.warn("Error %s", e.strerror)
<
< if start and not pr_enabled:
< pr = cProfile.Profile()
< pr.enable()
< pr_enabled = True
< LOG.warn("enabled cprofile")
< if not start and pr_enabled:
< pr.disable()
< pr.create_stats()
< pr.dump_stats("/tmp/%s"%fname)
< pr_enabled = False
< LOG.warn("disabled cprofile")
In polling loop:
< self.toggle_cprofile() |
|
| 2015-09-04 20:58:02 |
Ramu Ramamurthy |
description |
I used cProfile to profile neutron-ovs-agent (from neutron kilo 2015.1.0) as VMs are provisioned (see code sample below to reproduce).
I find a couple of functions in the IptablesManager scaling poorly with # of VMs (_modify_rules, and its callee find_last_entry).
As the # of current VMs doubles, the time spent in these functions to provision 10 new VMs also roughly doubles,
While we wait for the new IptablesTables firewall driver:
https://blueprints.launchpad.net/neutron/+spec/new-iptables-driver
Can we improve the performance of the current iptables firewall on those 2 functions which do a lot of string processing ?
Current: #VMs: 20, # SG rules: 657,
provision 10 new VMs
60 0.143 0.002 3.979 0.066 iptables_manager.py:511(_modify_rules)
25989 2.752 0.000 3.332 0.000 iptables_manager.py:504(_find_last_entry)
Current #VMs: 40, # SG rules: 1277 ,
provision 10 new VMs
65 0.220 0.003 7.974 0.123 iptables_manager.py:511(_modify_rules)
38891 5.782 0.000 6.986 0.000 iptables_manager.py:504(_find_last_entry)
Current #VMs: 80, # SG rules: 2517 ,
provision 10 new VMs
30 0.274 0.009 20.496 0.683 iptables_manager.py:511(_modify_rules)
43862 15.920 0.000 19.292 0.000 iptables_manager.py:504(_find_last_entry)
current #VMs: 160, #SG rules: 4997,
provision 10 new VMs
20 0.375 0.019 49.255 2.463 iptables_manager.py:511(_modify_rules)
56478 39.275 0.001 47.629 0.001 iptables_manager.py:504(_find_last_entry)
To Reproduce:
-------------------
Make following change to neutron_ovs_agent.py to enable/disable cProfile for a given scenario.
< import cProfile
< import os.path
< pr_enabled = False
< pr = None
In OVSNeutronAgent add method:
< def toggle_cprofile(self):
< global pr, pr_enabled
< start = False
< data = ""
< fname = "vm.profile"
< try:
< if os.path.isfile("/tmp/cprof"):
< start = True
< except IOError as e:
< LOG.warn("Error %s", e.strerror)
<
< if start and not pr_enabled:
< pr = cProfile.Profile()
< pr.enable()
< pr_enabled = True
< LOG.warn("enabled cprofile")
< if not start and pr_enabled:
< pr.disable()
< pr.create_stats()
< pr.dump_stats("/tmp/%s"%fname)
< pr_enabled = False
< LOG.warn("disabled cprofile")
In polling loop:
< self.toggle_cprofile() |
I used cProfile to profile neutron-ovs-agent (from neutron kilo 2015.1.0) as VMs are provisioned (see code sample below to reproduce).
I find a couple of functions in the IptablesManager scaling poorly with # of VMs (_modify_rules, and its callee find_last_entry).
As the # of current VMs doubles, the time spent in these functions to provision 10 new VMs also roughly doubles,
While we wait for the new IptablesTables firewall driver:
https://blueprints.launchpad.net/neutron/+spec/new-iptables-driver
Can we improve the performance of the current iptables firewall on those 2 functions which do a lot of string processing on iptables rule strings ?
Current: #VMs: 20, # SG rules: 657,
provision 10 new VMs
60 0.143 0.002 3.979 0.066 iptables_manager.py:511(_modify_rules)
25989 2.752 0.000 3.332 0.000 iptables_manager.py:504(_find_last_entry)
Current #VMs: 40, # SG rules: 1277 ,
provision 10 new VMs
65 0.220 0.003 7.974 0.123 iptables_manager.py:511(_modify_rules)
38891 5.782 0.000 6.986 0.000 iptables_manager.py:504(_find_last_entry)
Current #VMs: 80, # SG rules: 2517 ,
provision 10 new VMs
30 0.274 0.009 20.496 0.683 iptables_manager.py:511(_modify_rules)
43862 15.920 0.000 19.292 0.000 iptables_manager.py:504(_find_last_entry)
current #VMs: 160, #SG rules: 4997,
provision 10 new VMs
20 0.375 0.019 49.255 2.463 iptables_manager.py:511(_modify_rules)
56478 39.275 0.001 47.629 0.001 iptables_manager.py:504(_find_last_entry)
To Reproduce:
-------------------
Make following change to neutron_ovs_agent.py to enable/disable cProfile for a given scenario.
< import cProfile
< import os.path
< pr_enabled = False
< pr = None
In OVSNeutronAgent add method:
< def toggle_cprofile(self):
< global pr, pr_enabled
< start = False
< data = ""
< fname = "vm.profile"
< try:
< if os.path.isfile("/tmp/cprof"):
< start = True
< except IOError as e:
< LOG.warn("Error %s", e.strerror)
<
< if start and not pr_enabled:
< pr = cProfile.Profile()
< pr.enable()
< pr_enabled = True
< LOG.warn("enabled cprofile")
< if not start and pr_enabled:
< pr.disable()
< pr.create_stats()
< pr.dump_stats("/tmp/%s"%fname)
< pr_enabled = False
< LOG.warn("disabled cprofile")
In polling loop:
< self.toggle_cprofile() |
|
| 2015-09-04 20:58:40 |
Ramu Ramamurthy |
description |
I used cProfile to profile neutron-ovs-agent (from neutron kilo 2015.1.0) as VMs are provisioned (see code sample below to reproduce).
I find a couple of functions in the IptablesManager scaling poorly with # of VMs (_modify_rules, and its callee find_last_entry).
As the # of current VMs doubles, the time spent in these functions to provision 10 new VMs also roughly doubles,
While we wait for the new IptablesTables firewall driver:
https://blueprints.launchpad.net/neutron/+spec/new-iptables-driver
Can we improve the performance of the current iptables firewall on those 2 functions which do a lot of string processing on iptables rule strings ?
Current: #VMs: 20, # SG rules: 657,
provision 10 new VMs
60 0.143 0.002 3.979 0.066 iptables_manager.py:511(_modify_rules)
25989 2.752 0.000 3.332 0.000 iptables_manager.py:504(_find_last_entry)
Current #VMs: 40, # SG rules: 1277 ,
provision 10 new VMs
65 0.220 0.003 7.974 0.123 iptables_manager.py:511(_modify_rules)
38891 5.782 0.000 6.986 0.000 iptables_manager.py:504(_find_last_entry)
Current #VMs: 80, # SG rules: 2517 ,
provision 10 new VMs
30 0.274 0.009 20.496 0.683 iptables_manager.py:511(_modify_rules)
43862 15.920 0.000 19.292 0.000 iptables_manager.py:504(_find_last_entry)
current #VMs: 160, #SG rules: 4997,
provision 10 new VMs
20 0.375 0.019 49.255 2.463 iptables_manager.py:511(_modify_rules)
56478 39.275 0.001 47.629 0.001 iptables_manager.py:504(_find_last_entry)
To Reproduce:
-------------------
Make following change to neutron_ovs_agent.py to enable/disable cProfile for a given scenario.
< import cProfile
< import os.path
< pr_enabled = False
< pr = None
In OVSNeutronAgent add method:
< def toggle_cprofile(self):
< global pr, pr_enabled
< start = False
< data = ""
< fname = "vm.profile"
< try:
< if os.path.isfile("/tmp/cprof"):
< start = True
< except IOError as e:
< LOG.warn("Error %s", e.strerror)
<
< if start and not pr_enabled:
< pr = cProfile.Profile()
< pr.enable()
< pr_enabled = True
< LOG.warn("enabled cprofile")
< if not start and pr_enabled:
< pr.disable()
< pr.create_stats()
< pr.dump_stats("/tmp/%s"%fname)
< pr_enabled = False
< LOG.warn("disabled cprofile")
In polling loop:
< self.toggle_cprofile() |
I used cProfile to profile neutron-ovs-agent (from neutron kilo 2015.1.0) as VMs are provisioned (see code sample below to reproduce).
I find a couple of functions in the IptablesManager scaling poorly with # of VMs (_modify_rules, and its callee find_last_entry).
As the # of current VMs doubles, the time spent in these functions to provision 10 new VMs also roughly doubles,
While we wait for the new IptablesTables firewall driver:
https://blueprints.launchpad.net/neutron/+spec/new-iptables-driver
Can we improve the performance of the current iptables firewall on those 2 functions which do a lot of string processing on iptables rule strings checking for dups ?
Current: #VMs: 20, # SG rules: 657,
provision 10 new VMs
60 0.143 0.002 3.979 0.066 iptables_manager.py:511(_modify_rules)
25989 2.752 0.000 3.332 0.000 iptables_manager.py:504(_find_last_entry)
Current #VMs: 40, # SG rules: 1277 ,
provision 10 new VMs
65 0.220 0.003 7.974 0.123 iptables_manager.py:511(_modify_rules)
38891 5.782 0.000 6.986 0.000 iptables_manager.py:504(_find_last_entry)
Current #VMs: 80, # SG rules: 2517 ,
provision 10 new VMs
30 0.274 0.009 20.496 0.683 iptables_manager.py:511(_modify_rules)
43862 15.920 0.000 19.292 0.000 iptables_manager.py:504(_find_last_entry)
current #VMs: 160, #SG rules: 4997,
provision 10 new VMs
20 0.375 0.019 49.255 2.463 iptables_manager.py:511(_modify_rules)
56478 39.275 0.001 47.629 0.001 iptables_manager.py:504(_find_last_entry)
To Reproduce:
-------------------
Make following change to neutron_ovs_agent.py to enable/disable cProfile for a given scenario.
< import cProfile
< import os.path
< pr_enabled = False
< pr = None
In OVSNeutronAgent add method:
< def toggle_cprofile(self):
< global pr, pr_enabled
< start = False
< data = ""
< fname = "vm.profile"
< try:
< if os.path.isfile("/tmp/cprof"):
< start = True
< except IOError as e:
< LOG.warn("Error %s", e.strerror)
<
< if start and not pr_enabled:
< pr = cProfile.Profile()
< pr.enable()
< pr_enabled = True
< LOG.warn("enabled cprofile")
< if not start and pr_enabled:
< pr.disable()
< pr.create_stats()
< pr.dump_stats("/tmp/%s"%fname)
< pr_enabled = False
< LOG.warn("disabled cprofile")
In polling loop:
< self.toggle_cprofile() |
|
| 2015-09-04 21:02:00 |
Ramu Ramamurthy |
description |
I used cProfile to profile neutron-ovs-agent (from neutron kilo 2015.1.0) as VMs are provisioned (see code sample below to reproduce).
I find a couple of functions in the IptablesManager scaling poorly with # of VMs (_modify_rules, and its callee find_last_entry).
As the # of current VMs doubles, the time spent in these functions to provision 10 new VMs also roughly doubles,
While we wait for the new IptablesTables firewall driver:
https://blueprints.launchpad.net/neutron/+spec/new-iptables-driver
Can we improve the performance of the current iptables firewall on those 2 functions which do a lot of string processing on iptables rule strings checking for dups ?
Current: #VMs: 20, # SG rules: 657,
provision 10 new VMs
60 0.143 0.002 3.979 0.066 iptables_manager.py:511(_modify_rules)
25989 2.752 0.000 3.332 0.000 iptables_manager.py:504(_find_last_entry)
Current #VMs: 40, # SG rules: 1277 ,
provision 10 new VMs
65 0.220 0.003 7.974 0.123 iptables_manager.py:511(_modify_rules)
38891 5.782 0.000 6.986 0.000 iptables_manager.py:504(_find_last_entry)
Current #VMs: 80, # SG rules: 2517 ,
provision 10 new VMs
30 0.274 0.009 20.496 0.683 iptables_manager.py:511(_modify_rules)
43862 15.920 0.000 19.292 0.000 iptables_manager.py:504(_find_last_entry)
current #VMs: 160, #SG rules: 4997,
provision 10 new VMs
20 0.375 0.019 49.255 2.463 iptables_manager.py:511(_modify_rules)
56478 39.275 0.001 47.629 0.001 iptables_manager.py:504(_find_last_entry)
To Reproduce:
-------------------
Make following change to neutron_ovs_agent.py to enable/disable cProfile for a given scenario.
< import cProfile
< import os.path
< pr_enabled = False
< pr = None
In OVSNeutronAgent add method:
< def toggle_cprofile(self):
< global pr, pr_enabled
< start = False
< data = ""
< fname = "vm.profile"
< try:
< if os.path.isfile("/tmp/cprof"):
< start = True
< except IOError as e:
< LOG.warn("Error %s", e.strerror)
<
< if start and not pr_enabled:
< pr = cProfile.Profile()
< pr.enable()
< pr_enabled = True
< LOG.warn("enabled cprofile")
< if not start and pr_enabled:
< pr.disable()
< pr.create_stats()
< pr.dump_stats("/tmp/%s"%fname)
< pr_enabled = False
< LOG.warn("disabled cprofile")
In polling loop:
< self.toggle_cprofile() |
I used cProfile to profile neutron-ovs-agent (from neutron kilo 2015.1.0) as VMs are provisioned (see code sample below to reproduce).
I find a couple of functions in the IptablesManager scaling poorly with # of VMs (_modify_rules, and its callee find_last_entry).
As the # of current VMs doubles, the time spent in these functions to provision 10 new VMs also roughly doubles,
While we wait for the new IptablesTables firewall driver:
https://blueprints.launchpad.net/neutron/+spec/new-iptables-driver
Can we improve the performance of the current iptables firewall on those 2 functions which do a lot of string processing on iptables rule strings checking for dups ?
Current: #VMs: 20, # SG rules: 657,
provision 10 new VMs
ncalls tottime percall cumtime percall filename:lineno(function)
60 0.143 0.002 3.979 0.066 iptables_manager.py:511(_modify_rules)
25989 2.752 0.000 3.332 0.000 iptables_manager.py:504(_find_last_entry)
Current #VMs: 40, # SG rules: 1277 ,
provision 10 new VMs
65 0.220 0.003 7.974 0.123 iptables_manager.py:511(_modify_rules)
38891 5.782 0.000 6.986 0.000 iptables_manager.py:504(_find_last_entry)
Current #VMs: 80, # SG rules: 2517 ,
provision 10 new VMs
30 0.274 0.009 20.496 0.683 iptables_manager.py:511(_modify_rules)
43862 15.920 0.000 19.292 0.000 iptables_manager.py:504(_find_last_entry)
current #VMs: 160, #SG rules: 4997,
provision 10 new VMs
20 0.375 0.019 49.255 2.463 iptables_manager.py:511(_modify_rules)
56478 39.275 0.001 47.629 0.001 iptables_manager.py:504(_find_last_entry)
To Reproduce:
-------------------
Make following change to neutron_ovs_agent.py to enable/disable cProfile for a given scenario.
< import cProfile
< import os.path
< pr_enabled = False
< pr = None
In OVSNeutronAgent add method:
< def toggle_cprofile(self):
< global pr, pr_enabled
< start = False
< data = ""
< fname = "vm.profile"
< try:
< if os.path.isfile("/tmp/cprof"):
< start = True
< except IOError as e:
< LOG.warn("Error %s", e.strerror)
<
< if start and not pr_enabled:
< pr = cProfile.Profile()
< pr.enable()
< pr_enabled = True
< LOG.warn("enabled cprofile")
< if not start and pr_enabled:
< pr.disable()
< pr.create_stats()
< pr.dump_stats("/tmp/%s"%fname)
< pr_enabled = False
< LOG.warn("disabled cprofile")
In polling loop:
< self.toggle_cprofile() |
|
| 2015-09-04 21:02:33 |
Ramu Ramamurthy |
description |
I used cProfile to profile neutron-ovs-agent (from neutron kilo 2015.1.0) as VMs are provisioned (see code sample below to reproduce).
I find a couple of functions in the IptablesManager scaling poorly with # of VMs (_modify_rules, and its callee find_last_entry).
As the # of current VMs doubles, the time spent in these functions to provision 10 new VMs also roughly doubles,
While we wait for the new IptablesTables firewall driver:
https://blueprints.launchpad.net/neutron/+spec/new-iptables-driver
Can we improve the performance of the current iptables firewall on those 2 functions which do a lot of string processing on iptables rule strings checking for dups ?
Current: #VMs: 20, # SG rules: 657,
provision 10 new VMs
ncalls tottime percall cumtime percall filename:lineno(function)
60 0.143 0.002 3.979 0.066 iptables_manager.py:511(_modify_rules)
25989 2.752 0.000 3.332 0.000 iptables_manager.py:504(_find_last_entry)
Current #VMs: 40, # SG rules: 1277 ,
provision 10 new VMs
65 0.220 0.003 7.974 0.123 iptables_manager.py:511(_modify_rules)
38891 5.782 0.000 6.986 0.000 iptables_manager.py:504(_find_last_entry)
Current #VMs: 80, # SG rules: 2517 ,
provision 10 new VMs
30 0.274 0.009 20.496 0.683 iptables_manager.py:511(_modify_rules)
43862 15.920 0.000 19.292 0.000 iptables_manager.py:504(_find_last_entry)
current #VMs: 160, #SG rules: 4997,
provision 10 new VMs
20 0.375 0.019 49.255 2.463 iptables_manager.py:511(_modify_rules)
56478 39.275 0.001 47.629 0.001 iptables_manager.py:504(_find_last_entry)
To Reproduce:
-------------------
Make following change to neutron_ovs_agent.py to enable/disable cProfile for a given scenario.
< import cProfile
< import os.path
< pr_enabled = False
< pr = None
In OVSNeutronAgent add method:
< def toggle_cprofile(self):
< global pr, pr_enabled
< start = False
< data = ""
< fname = "vm.profile"
< try:
< if os.path.isfile("/tmp/cprof"):
< start = True
< except IOError as e:
< LOG.warn("Error %s", e.strerror)
<
< if start and not pr_enabled:
< pr = cProfile.Profile()
< pr.enable()
< pr_enabled = True
< LOG.warn("enabled cprofile")
< if not start and pr_enabled:
< pr.disable()
< pr.create_stats()
< pr.dump_stats("/tmp/%s"%fname)
< pr_enabled = False
< LOG.warn("disabled cprofile")
In polling loop:
< self.toggle_cprofile() |
I used cProfile to profile neutron-ovs-agent (from neutron kilo 2015.1.0) as VMs are provisioned (see code sample below to reproduce).
I find a couple of functions in the IptablesManager scaling poorly with # of VMs (_modify_rules, and its callee find_last_entry).
As the # of current VMs doubles, the time spent in these functions to provision 10 new VMs also roughly doubles,
While we wait for the new IptablesTables firewall driver:
https://blueprints.launchpad.net/neutron/+spec/new-iptables-driver
Can we improve the performance of the current iptables firewall on those 2 functions which do a lot of string processing on iptables rule strings checking for dups ?
Current: #VMs: 20, # SG rules: 657,
provision 10 new VMs
ncalls tottime percall cumtime percall filename:lineno(function)
60 0.143 0.002 3.979 0.066 iptables_manager.py:511(_modify_rules)
25989 2.752 0.000 3.332 0.000 iptables_manager.py:504(_find_last_entry)
Current #VMs: 40, # SG rules: 1277 ,
provision 10 new VMs
65 0.220 0.003 7.974 0.123 iptables_manager.py:511(_modify_rules)
38891 5.782 0.000 6.986 0.000 iptables_manager.py:504(_find_last_entry)
Current #VMs: 80, # SG rules: 2517 ,
provision 10 new VMs
30 0.274 0.009 20.496 0.683 iptables_manager.py:511(_modify_rules)
43862 15.920 0.000 19.292 0.000 iptables_manager.py:504(_find_last_entry)
current #VMs: 160, #SG rules: 4997,
provision 10 new VMs
20 0.375 0.019 49.255 2.463 iptables_manager.py:511(_modify_rules)
56478 39.275 0.001 47.629 0.001 iptables_manager.py:504(_find_last_entry)
To Reproduce:
-------------------
Make following change to neutron_ovs_agent.py to enable/disable cProfile for a given scenario.
< import cProfile
< import os.path
< pr_enabled = False
< pr = None
In OVSNeutronAgent add method:
< def toggle_cprofile(self):
< global pr, pr_enabled
< start = False
< data = ""
< fname = "vm.profile"
< try:
< if os.path.isfile("/tmp/cprof"):
< start = True
< except IOError as e:
< LOG.warn("Error %s", e.strerror)
<
< if start and not pr_enabled:
< pr = cProfile.Profile()
< pr.enable()
< pr_enabled = True
< LOG.warn("enabled cprofile")
< if not start and pr_enabled:
< pr.disable()
< pr.create_stats()
< pr.dump_stats("/tmp/%s"%fname)
< pr_enabled = False
< LOG.warn("disabled cprofile")
In polling loop:
< self.toggle_cprofile() |
|
| 2015-09-04 21:03:59 |
Ramu Ramamurthy |
description |
I used cProfile to profile neutron-ovs-agent (from neutron kilo 2015.1.0) as VMs are provisioned (see code sample below to reproduce).
I find a couple of functions in the IptablesManager scaling poorly with # of VMs (_modify_rules, and its callee find_last_entry).
As the # of current VMs doubles, the time spent in these functions to provision 10 new VMs also roughly doubles,
While we wait for the new IptablesTables firewall driver:
https://blueprints.launchpad.net/neutron/+spec/new-iptables-driver
Can we improve the performance of the current iptables firewall on those 2 functions which do a lot of string processing on iptables rule strings checking for dups ?
Current: #VMs: 20, # SG rules: 657,
provision 10 new VMs
ncalls tottime percall cumtime percall filename:lineno(function)
60 0.143 0.002 3.979 0.066 iptables_manager.py:511(_modify_rules)
25989 2.752 0.000 3.332 0.000 iptables_manager.py:504(_find_last_entry)
Current #VMs: 40, # SG rules: 1277 ,
provision 10 new VMs
65 0.220 0.003 7.974 0.123 iptables_manager.py:511(_modify_rules)
38891 5.782 0.000 6.986 0.000 iptables_manager.py:504(_find_last_entry)
Current #VMs: 80, # SG rules: 2517 ,
provision 10 new VMs
30 0.274 0.009 20.496 0.683 iptables_manager.py:511(_modify_rules)
43862 15.920 0.000 19.292 0.000 iptables_manager.py:504(_find_last_entry)
current #VMs: 160, #SG rules: 4997,
provision 10 new VMs
20 0.375 0.019 49.255 2.463 iptables_manager.py:511(_modify_rules)
56478 39.275 0.001 47.629 0.001 iptables_manager.py:504(_find_last_entry)
To Reproduce:
-------------------
Make following change to neutron_ovs_agent.py to enable/disable cProfile for a given scenario.
< import cProfile
< import os.path
< pr_enabled = False
< pr = None
In OVSNeutronAgent add method:
< def toggle_cprofile(self):
< global pr, pr_enabled
< start = False
< data = ""
< fname = "vm.profile"
< try:
< if os.path.isfile("/tmp/cprof"):
< start = True
< except IOError as e:
< LOG.warn("Error %s", e.strerror)
<
< if start and not pr_enabled:
< pr = cProfile.Profile()
< pr.enable()
< pr_enabled = True
< LOG.warn("enabled cprofile")
< if not start and pr_enabled:
< pr.disable()
< pr.create_stats()
< pr.dump_stats("/tmp/%s"%fname)
< pr_enabled = False
< LOG.warn("disabled cprofile")
In polling loop:
< self.toggle_cprofile() |
I used cProfile to profile neutron-ovs-agent (from neutron kilo 2015.1.0) as VMs are provisioned (see code sample below to reproduce).
I find a couple of functions in the IptablesManager scaling poorly with # of VMs (_modify_rules, and its callee find_last_entry).
As the # of current VMs doubles, the time spent in these functions to provision 10 new VMs also roughly doubles,
While we wait for the new IptablesTables firewall driver:
https://blueprints.launchpad.net/neutron/+spec/new-iptables-driver
Can we improve the performance of the current iptables firewall on those 2 functions which do a lot of string processing on iptables rule strings checking for dups ?
Current: #VMs: 20, # SG rules: 657,
provision 10 new VMs
ncalls tottime percall cumtime percall filename:lineno(function)
60 0.143 0.002 3.979 0.066 iptables_manager.py:511(_modify_rules)
25989 2.752 0.000 3.332 0.000 iptables_manager.py:504(_find_last_entry)
Current #VMs: 40, # SG rules: 1277 ,
provision 10 new VMs
65 0.220 0.003 7.974 0.123 iptables_manager.py:511(_modify_rules)
38891 5.782 0.000 6.986 0.000 iptables_manager.py:504(_find_last_entry)
Current #VMs: 80, # SG rules: 2517 ,
provision 10 new VMs
30 0.274 0.009 20.496 0.683 iptables_manager.py:511(_modify_rules)
43862 15.920 0.000 19.292 0.000 iptables_manager.py:504(_find_last_entry)
current #VMs: 160, #SG rules: 4997,
provision 10 new VMs
20 0.375 0.019 49.255 2.463 iptables_manager.py:511(_modify_rules)
56478 39.275 0.001 47.629 0.001 iptables_manager.py:504(_find_last_entry)
To Reproduce:
-------------------
Make following change to neutron_ovs_agent.py to enable/disable cProfile for a given scenario.
import cProfile
import os.path
pr_enabled = False
pr = None
In OVSNeutronAgent add method:
def toggle_cprofile(self):
global pr, pr_enabled
start = False
data = ""
fname = "vm.profile"
try:
if os.path.isfile("/tmp/cprof"):
start = True
except IOError as e:
LOG.warn("Error %s", e.strerror)
if start and not pr_enabled:
pr = cProfile.Profile()
pr.enable()
pr_enabled = True
LOG.warn("enabled cprofile")
if not start and pr_enabled:
pr.disable()
pr.create_stats()
pr.dump_stats("/tmp/%s"%fname)
pr_enabled = False
LOG.warn("disabled cprofile")
In polling loop:
self.toggle_cprofile() |
|
| 2015-09-04 21:04:41 |
Ramu Ramamurthy |
description |
I used cProfile to profile neutron-ovs-agent (from neutron kilo 2015.1.0) as VMs are provisioned (see code sample below to reproduce).
I find a couple of functions in the IptablesManager scaling poorly with # of VMs (_modify_rules, and its callee find_last_entry).
As the # of current VMs doubles, the time spent in these functions to provision 10 new VMs also roughly doubles,
While we wait for the new IptablesTables firewall driver:
https://blueprints.launchpad.net/neutron/+spec/new-iptables-driver
Can we improve the performance of the current iptables firewall on those 2 functions which do a lot of string processing on iptables rule strings checking for dups ?
Current: #VMs: 20, # SG rules: 657,
provision 10 new VMs
ncalls tottime percall cumtime percall filename:lineno(function)
60 0.143 0.002 3.979 0.066 iptables_manager.py:511(_modify_rules)
25989 2.752 0.000 3.332 0.000 iptables_manager.py:504(_find_last_entry)
Current #VMs: 40, # SG rules: 1277 ,
provision 10 new VMs
65 0.220 0.003 7.974 0.123 iptables_manager.py:511(_modify_rules)
38891 5.782 0.000 6.986 0.000 iptables_manager.py:504(_find_last_entry)
Current #VMs: 80, # SG rules: 2517 ,
provision 10 new VMs
30 0.274 0.009 20.496 0.683 iptables_manager.py:511(_modify_rules)
43862 15.920 0.000 19.292 0.000 iptables_manager.py:504(_find_last_entry)
current #VMs: 160, #SG rules: 4997,
provision 10 new VMs
20 0.375 0.019 49.255 2.463 iptables_manager.py:511(_modify_rules)
56478 39.275 0.001 47.629 0.001 iptables_manager.py:504(_find_last_entry)
To Reproduce:
-------------------
Make following change to neutron_ovs_agent.py to enable/disable cProfile for a given scenario.
import cProfile
import os.path
pr_enabled = False
pr = None
In OVSNeutronAgent add method:
def toggle_cprofile(self):
global pr, pr_enabled
start = False
data = ""
fname = "vm.profile"
try:
if os.path.isfile("/tmp/cprof"):
start = True
except IOError as e:
LOG.warn("Error %s", e.strerror)
if start and not pr_enabled:
pr = cProfile.Profile()
pr.enable()
pr_enabled = True
LOG.warn("enabled cprofile")
if not start and pr_enabled:
pr.disable()
pr.create_stats()
pr.dump_stats("/tmp/%s"%fname)
pr_enabled = False
LOG.warn("disabled cprofile")
In polling loop:
self.toggle_cprofile() |
I used cProfile to profile neutron-ovs-agent (from neutron kilo 2015.1.0) as VMs are provisioned (see code sample below to reproduce).
I find a couple of functions in the IptablesManager scaling poorly with # of VMs (_modify_rules, and its callee find_last_entry).
As the # of current VMs doubles, the time spent in these functions to provision 10 new VMs also roughly doubles,
While we wait for the new IptablesTables firewall driver:
https://blueprints.launchpad.net/neutron/+spec/new-iptables-driver
Can we improve the performance of the current iptables firewall on those 2 functions which do a lot of string processing on iptables rule strings checking for dups ?
Current: #VMs: 20, # SG rules: 657,
provision 10 new VMs
ncalls tottime percall cumtime percall filename:lineno(function)
60 0.143 0.002 3.979 0.066 iptables_manager.py:511(_modify_rules)
25989 2.752 0.000 3.332 0.000 iptables_manager.py:504(_find_last_entry)
Current #VMs: 40, # SG rules: 1277 ,
provision 10 new VMs
65 0.220 0.003 7.974 0.123 iptables_manager.py:511(_modify_rules)
38891 5.782 0.000 6.986 0.000 iptables_manager.py:504(_find_last_entry)
Current #VMs: 80, # SG rules: 2517 ,
provision 10 new VMs
30 0.274 0.009 20.496 0.683 iptables_manager.py:511(_modify_rules)
43862 15.920 0.000 19.292 0.000 iptables_manager.py:504(_find_last_entry)
current #VMs: 160, #SG rules: 4997,
provision 10 new VMs
20 0.375 0.019 49.255 2.463 iptables_manager.py:511(_modify_rules)
56478 39.275 0.001 47.629 0.001 iptables_manager.py:504(_find_last_entry)
To Reproduce:
-------------------
Make following change to neutron_ovs_agent.py to enable/disable cProfile for a given scenario.
import cProfile
import os.path
pr_enabled = False
pr = None
In OVSNeutronAgent add method:
def toggle_cprofile(self):
global pr, pr_enabled
start = False
data = ""
fname = "vm.profile"
try:
if os.path.isfile("/tmp/cprof"):
start = True
except IOError as e:
LOG.warn("Error %s", e.strerror)
if start and not pr_enabled:
pr = cProfile.Profile()
pr.enable()
pr_enabled = True
LOG.warn("enabled cprofile")
if not start and pr_enabled:
pr.disable()
pr.create_stats()
pr.dump_stats("/tmp/%s"%fname)
pr_enabled = False
LOG.warn("disabled cprofile")
In polling loop:
self.toggle_cprofile() |
|
| 2015-09-04 22:02:49 |
Ramu Ramamurthy |
description |
I used cProfile to profile neutron-ovs-agent (from neutron kilo 2015.1.0) as VMs are provisioned (see code sample below to reproduce).
I find a couple of functions in the IptablesManager scaling poorly with # of VMs (_modify_rules, and its callee find_last_entry).
As the # of current VMs doubles, the time spent in these functions to provision 10 new VMs also roughly doubles,
While we wait for the new IptablesTables firewall driver:
https://blueprints.launchpad.net/neutron/+spec/new-iptables-driver
Can we improve the performance of the current iptables firewall on those 2 functions which do a lot of string processing on iptables rule strings checking for dups ?
Current: #VMs: 20, # SG rules: 657,
provision 10 new VMs
ncalls tottime percall cumtime percall filename:lineno(function)
60 0.143 0.002 3.979 0.066 iptables_manager.py:511(_modify_rules)
25989 2.752 0.000 3.332 0.000 iptables_manager.py:504(_find_last_entry)
Current #VMs: 40, # SG rules: 1277 ,
provision 10 new VMs
65 0.220 0.003 7.974 0.123 iptables_manager.py:511(_modify_rules)
38891 5.782 0.000 6.986 0.000 iptables_manager.py:504(_find_last_entry)
Current #VMs: 80, # SG rules: 2517 ,
provision 10 new VMs
30 0.274 0.009 20.496 0.683 iptables_manager.py:511(_modify_rules)
43862 15.920 0.000 19.292 0.000 iptables_manager.py:504(_find_last_entry)
current #VMs: 160, #SG rules: 4997,
provision 10 new VMs
20 0.375 0.019 49.255 2.463 iptables_manager.py:511(_modify_rules)
56478 39.275 0.001 47.629 0.001 iptables_manager.py:504(_find_last_entry)
To Reproduce:
-------------------
Make following change to neutron_ovs_agent.py to enable/disable cProfile for a given scenario.
import cProfile
import os.path
pr_enabled = False
pr = None
In OVSNeutronAgent add method:
def toggle_cprofile(self):
global pr, pr_enabled
start = False
data = ""
fname = "vm.profile"
try:
if os.path.isfile("/tmp/cprof"):
start = True
except IOError as e:
LOG.warn("Error %s", e.strerror)
if start and not pr_enabled:
pr = cProfile.Profile()
pr.enable()
pr_enabled = True
LOG.warn("enabled cprofile")
if not start and pr_enabled:
pr.disable()
pr.create_stats()
pr.dump_stats("/tmp/%s"%fname)
pr_enabled = False
LOG.warn("disabled cprofile")
In polling loop:
self.toggle_cprofile() |
I used cProfile to profile neutron-ovs-agent (from neutron kilo 2015.1.0) as VMs are provisioned (see code sample below to reproduce).
I find a couple of functions in the IptablesManager scaling poorly with # of VMs (_modify_rules, and its callee find_last_entry).
As the # of current VMs doubles, the time spent in these functions to provision 10 new VMs also roughly doubles,
While we wait for the new IptablesTables firewall driver:
https://blueprints.launchpad.net/neutron/+spec/new-iptables-driver
Can we improve the performance of the current iptables firewall on those 2 functions which do a lot of string processing on iptables rule strings checking for dups ?
Current: #VMs: 20, # SG rules: 657,
provision 10 new VMs
ncalls tottime percall cumtime percall filename:lineno(function)
60 0.143 0.002 3.979 0.066 iptables_manager.py:511(_modify_rules)
25989 2.752 0.000 3.332 0.000 iptables_manager.py:504(_find_last_entry)
Cumulative time spent in _find_last_entry: 3.3 sec
Current #VMs: 40, # SG rules: 1277 ,
provision 10 new VMs
65 0.220 0.003 7.974 0.123 iptables_manager.py:511(_modify_rules)
38891 5.782 0.000 6.986 0.000 iptables_manager.py:504(_find_last_entry)
Cumulative time spent in _find_last_entry: 6.9 sec
Current #VMs: 80, # SG rules: 2517 ,
provision 10 new VMs
30 0.274 0.009 20.496 0.683 iptables_manager.py:511(_modify_rules)
43862 15.920 0.000 19.292 0.000 iptables_manager.py:504(_find_last_entry)
Cumulative time spent in _find_last_entry: 19.2 sec
current #VMs: 160, #SG rules: 4997,
provision 10 new VMs
20 0.375 0.019 49.255 2.463 iptables_manager.py:511(_modify_rules)
56478 39.275 0.001 47.629 0.001 iptables_manager.py:504(_find_last_entry)
Cumulative time spent in _find_last_entry: 47.6 sec
To Reproduce:
-------------------
Make following change to neutron_ovs_agent.py to enable/disable cProfile for a given scenario.
import cProfile
import os.path
pr_enabled = False
pr = None
In OVSNeutronAgent add method:
def toggle_cprofile(self):
global pr, pr_enabled
start = False
data = ""
fname = "vm.profile"
try:
if os.path.isfile("/tmp/cprof"):
start = True
except IOError as e:
LOG.warn("Error %s", e.strerror)
if start and not pr_enabled:
pr = cProfile.Profile()
pr.enable()
pr_enabled = True
LOG.warn("enabled cprofile")
if not start and pr_enabled:
pr.disable()
pr.create_stats()
pr.dump_stats("/tmp/%s"%fname)
pr_enabled = False
LOG.warn("disabled cprofile")
In polling loop:
self.toggle_cprofile() |
|
| 2015-09-08 17:37:54 |
Kyle Mestery |
neutron: assignee |
|
Miguel Angel Ajo (mangelajo) |
|
| 2015-09-08 17:37:59 |
Kyle Mestery |
neutron: status |
New |
Triaged |
|
| 2015-09-08 17:38:14 |
Kyle Mestery |
neutron: importance |
Undecided |
High |
|
| 2015-09-08 18:10:42 |
Ramu Ramamurthy |
description |
I used cProfile to profile neutron-ovs-agent (from neutron kilo 2015.1.0) as VMs are provisioned (see code sample below to reproduce).
I find a couple of functions in the IptablesManager scaling poorly with # of VMs (_modify_rules, and its callee find_last_entry).
As the # of current VMs doubles, the time spent in these functions to provision 10 new VMs also roughly doubles,
While we wait for the new IptablesTables firewall driver:
https://blueprints.launchpad.net/neutron/+spec/new-iptables-driver
Can we improve the performance of the current iptables firewall on those 2 functions which do a lot of string processing on iptables rule strings checking for dups ?
Current: #VMs: 20, # SG rules: 657,
provision 10 new VMs
ncalls tottime percall cumtime percall filename:lineno(function)
60 0.143 0.002 3.979 0.066 iptables_manager.py:511(_modify_rules)
25989 2.752 0.000 3.332 0.000 iptables_manager.py:504(_find_last_entry)
Cumulative time spent in _find_last_entry: 3.3 sec
Current #VMs: 40, # SG rules: 1277 ,
provision 10 new VMs
65 0.220 0.003 7.974 0.123 iptables_manager.py:511(_modify_rules)
38891 5.782 0.000 6.986 0.000 iptables_manager.py:504(_find_last_entry)
Cumulative time spent in _find_last_entry: 6.9 sec
Current #VMs: 80, # SG rules: 2517 ,
provision 10 new VMs
30 0.274 0.009 20.496 0.683 iptables_manager.py:511(_modify_rules)
43862 15.920 0.000 19.292 0.000 iptables_manager.py:504(_find_last_entry)
Cumulative time spent in _find_last_entry: 19.2 sec
current #VMs: 160, #SG rules: 4997,
provision 10 new VMs
20 0.375 0.019 49.255 2.463 iptables_manager.py:511(_modify_rules)
56478 39.275 0.001 47.629 0.001 iptables_manager.py:504(_find_last_entry)
Cumulative time spent in _find_last_entry: 47.6 sec
To Reproduce:
-------------------
Make following change to neutron_ovs_agent.py to enable/disable cProfile for a given scenario.
import cProfile
import os.path
pr_enabled = False
pr = None
In OVSNeutronAgent add method:
def toggle_cprofile(self):
global pr, pr_enabled
start = False
data = ""
fname = "vm.profile"
try:
if os.path.isfile("/tmp/cprof"):
start = True
except IOError as e:
LOG.warn("Error %s", e.strerror)
if start and not pr_enabled:
pr = cProfile.Profile()
pr.enable()
pr_enabled = True
LOG.warn("enabled cprofile")
if not start and pr_enabled:
pr.disable()
pr.create_stats()
pr.dump_stats("/tmp/%s"%fname)
pr_enabled = False
LOG.warn("disabled cprofile")
In polling loop:
self.toggle_cprofile() |
I used cProfile to profile neutron-ovs-agent (from neutron kilo 2015.1.0) as VMs are provisioned (see code sample below to reproduce).
I find a couple of functions in the IptablesManager scaling poorly with # of VMs (_modify_rules, and its callee find_last_entry).
As the # of current VMs doubles, the time spent in these functions to provision 10 new VMs also roughly doubles,
While we wait for the new IptablesTables firewall driver:
https://blueprints.launchpad.net/neutron/+spec/new-iptables-driver
Can we improve the performance of the current iptables firewall on those 2 functions which do a lot of string processing on iptables rule strings checking for dups ?
Current: #VMs: 20, # SG rules: 657,
provision 10 new VMs
ncalls tottime percall cumtime percall filename:lineno(function)
60 0.143 0.002 3.979 0.066 iptables_manager.py:511(_modify_rules)
25989 2.752 0.000 3.332 0.000 iptables_manager.py:504(_find_last_entry)
Cumulative time spent in _find_last_entry: 3.3 sec
Current #VMs: 40, # SG rules: 1277 ,
provision 10 new VMs
65 0.220 0.003 7.974 0.123 iptables_manager.py:511(_modify_rules)
38891 5.782 0.000 6.986 0.000 iptables_manager.py:504(_find_last_entry)
Cumulative time spent in _find_last_entry: 6.9 sec
Current #VMs: 80, # SG rules: 2517 ,
provision 10 new VMs
30 0.274 0.009 20.496 0.683 iptables_manager.py:511(_modify_rules)
43862 15.920 0.000 19.292 0.000 iptables_manager.py:504(_find_last_entry)
Cumulative time spent in _find_last_entry: 19.2 sec
current #VMs: 160, #SG rules: 4997,
provision 10 new VMs
20 0.375 0.019 49.255 2.463 iptables_manager.py:511(_modify_rules)
56478 39.275 0.001 47.629 0.001 iptables_manager.py:504(_find_last_entry)
Cumulative time spent in _find_last_entry: 47.6 sec
--------------------
To Reproduce: THis is one way where we can control start/stop of profiling
based on presence of a file (/tmp/cprof) in the file-system
-------
Make following change to neutron_ovs_agent.py to enable/disable cProfile for a given scenario.
import cProfile
import os.path
pr_enabled = False
pr = None
In OVSNeutronAgent add method:
def toggle_cprofile(self):
global pr, pr_enabled
start = False
data = ""
fname = "vm.profile"
try:
if os.path.isfile("/tmp/cprof"):
start = True
except IOError as e:
LOG.warn("Error %s", e.strerror)
if start and not pr_enabled:
pr = cProfile.Profile()
pr.enable()
pr_enabled = True
LOG.warn("enabled cprofile")
if not start and pr_enabled:
pr.disable()
pr.create_stats()
pr.dump_stats("/tmp/%s"%fname)
pr_enabled = False
LOG.warn("disabled cprofile")
In polling loop:
self.toggle_cprofile()
-------------
This is another way to run the cProfile, but here, there is no way to control start/stop of profiling, and the profile includes the initialization also.
Run, neutron-ovs-agent as follows:
sudo -u neutron bash -c "/usr/bin/python -m cProfile -o /tmp/vm_1.profile /usr/bin/neutron-openvswitch-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini --log-file /var/log/neutron/openvswitch-agent.log" |
|
| 2015-09-09 04:49:55 |
Ramu Ramamurthy |
description |
I used cProfile to profile neutron-ovs-agent (from neutron kilo 2015.1.0) as VMs are provisioned (see code sample below to reproduce).
I find a couple of functions in the IptablesManager scaling poorly with # of VMs (_modify_rules, and its callee find_last_entry).
As the # of current VMs doubles, the time spent in these functions to provision 10 new VMs also roughly doubles,
While we wait for the new IptablesTables firewall driver:
https://blueprints.launchpad.net/neutron/+spec/new-iptables-driver
Can we improve the performance of the current iptables firewall on those 2 functions which do a lot of string processing on iptables rule strings checking for dups ?
Current: #VMs: 20, # SG rules: 657,
provision 10 new VMs
ncalls tottime percall cumtime percall filename:lineno(function)
60 0.143 0.002 3.979 0.066 iptables_manager.py:511(_modify_rules)
25989 2.752 0.000 3.332 0.000 iptables_manager.py:504(_find_last_entry)
Cumulative time spent in _find_last_entry: 3.3 sec
Current #VMs: 40, # SG rules: 1277 ,
provision 10 new VMs
65 0.220 0.003 7.974 0.123 iptables_manager.py:511(_modify_rules)
38891 5.782 0.000 6.986 0.000 iptables_manager.py:504(_find_last_entry)
Cumulative time spent in _find_last_entry: 6.9 sec
Current #VMs: 80, # SG rules: 2517 ,
provision 10 new VMs
30 0.274 0.009 20.496 0.683 iptables_manager.py:511(_modify_rules)
43862 15.920 0.000 19.292 0.000 iptables_manager.py:504(_find_last_entry)
Cumulative time spent in _find_last_entry: 19.2 sec
current #VMs: 160, #SG rules: 4997,
provision 10 new VMs
20 0.375 0.019 49.255 2.463 iptables_manager.py:511(_modify_rules)
56478 39.275 0.001 47.629 0.001 iptables_manager.py:504(_find_last_entry)
Cumulative time spent in _find_last_entry: 47.6 sec
--------------------
To Reproduce: THis is one way where we can control start/stop of profiling
based on presence of a file (/tmp/cprof) in the file-system
-------
Make following change to neutron_ovs_agent.py to enable/disable cProfile for a given scenario.
import cProfile
import os.path
pr_enabled = False
pr = None
In OVSNeutronAgent add method:
def toggle_cprofile(self):
global pr, pr_enabled
start = False
data = ""
fname = "vm.profile"
try:
if os.path.isfile("/tmp/cprof"):
start = True
except IOError as e:
LOG.warn("Error %s", e.strerror)
if start and not pr_enabled:
pr = cProfile.Profile()
pr.enable()
pr_enabled = True
LOG.warn("enabled cprofile")
if not start and pr_enabled:
pr.disable()
pr.create_stats()
pr.dump_stats("/tmp/%s"%fname)
pr_enabled = False
LOG.warn("disabled cprofile")
In polling loop:
self.toggle_cprofile()
-------------
This is another way to run the cProfile, but here, there is no way to control start/stop of profiling, and the profile includes the initialization also.
Run, neutron-ovs-agent as follows:
sudo -u neutron bash -c "/usr/bin/python -m cProfile -o /tmp/vm_1.profile /usr/bin/neutron-openvswitch-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini --log-file /var/log/neutron/openvswitch-agent.log" |
I used cProfile to profile neutron-ovs-agent (from neutron kilo 2015.1.0) as VMs are provisioned (see code sample below to reproduce).
I find a couple of functions in the IptablesManager scaling poorly with # of VMs (_modify_rules, and its callee find_last_entry).
As the # of current VMs doubles, the time spent in these functions to provision 10 new VMs also roughly doubles,
While we wait for the new IptablesTables firewall driver:
https://blueprints.launchpad.net/neutron/+spec/new-iptables-driver
Can we improve the performance of the current iptables firewall on those 2 functions which do a lot of string processing on iptables rule strings checking for dups ?
Current: #VMs: 20, # iptables rules: 657,
provision 10 new VMs
ncalls tottime percall cumtime percall filename:lineno(function)
60 0.143 0.002 3.979 0.066 iptables_manager.py:511(_modify_rules)
25989 2.752 0.000 3.332 0.000 iptables_manager.py:504(_find_last_entry)
Cumulative time spent in _find_last_entry: 3.3 sec
Current #VMs: 40, # iptables rules: 1277 ,
provision 10 new VMs
65 0.220 0.003 7.974 0.123 iptables_manager.py:511(_modify_rules)
38891 5.782 0.000 6.986 0.000 iptables_manager.py:504(_find_last_entry)
Cumulative time spent in _find_last_entry: 6.9 sec
Current #VMs: 80, # iptables rules: 2517 ,
provision 10 new VMs
30 0.274 0.009 20.496 0.683 iptables_manager.py:511(_modify_rules)
43862 15.920 0.000 19.292 0.000 iptables_manager.py:504(_find_last_entry)
Cumulative time spent in _find_last_entry: 19.2 sec
current #VMs: 160, # iptables rules: 4997,
provision 10 new VMs
20 0.375 0.019 49.255 2.463 iptables_manager.py:511(_modify_rules)
56478 39.275 0.001 47.629 0.001 iptables_manager.py:504(_find_last_entry)
Cumulative time spent in _find_last_entry: 47.6 sec
--------------------
To Reproduce: THis is one way where we can control start/stop of profiling
based on presence of a file (/tmp/cprof) in the file-system
-------
Make following change to neutron_ovs_agent.py to enable/disable cProfile for a given scenario.
import cProfile
import os.path
pr_enabled = False
pr = None
In OVSNeutronAgent add method:
def toggle_cprofile(self):
global pr, pr_enabled
start = False
data = ""
fname = "vm.profile"
try:
if os.path.isfile("/tmp/cprof"):
start = True
except IOError as e:
LOG.warn("Error %s", e.strerror)
if start and not pr_enabled:
pr = cProfile.Profile()
pr.enable()
pr_enabled = True
LOG.warn("enabled cprofile")
if not start and pr_enabled:
pr.disable()
pr.create_stats()
pr.dump_stats("/tmp/%s"%fname)
pr_enabled = False
LOG.warn("disabled cprofile")
In polling loop:
self.toggle_cprofile()
-------------
This is another way to run the cProfile, but here, there is no way to control start/stop of profiling, and the profile includes the initialization also.
Run, neutron-ovs-agent as follows:
sudo -u neutron bash -c "/usr/bin/python -m cProfile -o /tmp/vm_1.profile /usr/bin/neutron-openvswitch-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini --log-file /var/log/neutron/openvswitch-agent.log" |
|
| 2015-09-10 17:07:07 |
Ramu Ramamurthy |
bug |
|
|
added subscriber Ryan Moats |
| 2015-09-11 22:40:44 |
Brian Haley |
bug |
|
|
added subscriber Brian Haley |
| 2015-10-02 09:52:57 |
Miguel Angel Ajo |
neutron: assignee |
Miguel Angel Ajo (mangelajo) |
|
|
| 2015-10-02 18:12:43 |
Ramu Ramamurthy |
neutron: status |
Triaged |
Fix Released |
|
| 2015-10-14 17:31:40 |
Armando Migliaccio |
marked as duplicate |
|
1502297 |
|
