[RFE] Add option(s) to enable profiling of agents

[Ramu Ramamurthy]

For motivation see this bug, where this "rfe" was requested.

https://bugs.launchpad.net/neutron/+bug/1502297

By running the neutron-agents under a profiler like cprofile, we can obtain valuable insights into the performance of the agent-code, and identify areas for improvement (see bug above for concrete example)

This bug seeks to add a config option in the agent config file like:

profiled_mode = false

by default the above is false, but when set to true, it starts the agent under cprofile, and dumps the profiled output to a file.

[Ihar Hrachyshka (ihar-hrachyshka)]

I don't think cProfile behaves correctly with eventlet.

[Kyle Mestery (mestery)]

Until Ihar's concerns are corrected, I don't see this one moving forward.

[Ramu Ramamurthy (ramu-ramamurthy)]

cprofile doesNOT appear to understand greenthread context switches. For instance, the called functions of plug_vifs - see below - appear to include greenthread functions which it should not.

Also, this has been discussed earlier (a while ago):
https://lists.secondlife.com/pipermail/eventletdev/2012-September/001094.html

As a final note, there appear to be profilers that are greenthread aware.. which could be considered if someone has the time to experiment with them.
http://pydoc.net/Python/eventlet/0.9.12/eventlet.green.profile/

Function called...
                                                                                 ncalls tottime cumtime
/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:598(plug_vifs) ->

                                                                                     1 0.000 0.000 /usr/lib/python2.7/site-packages/eventlet/greenthread.py:18(sleep)
                                                                                      1 0.000 0.000 /usr/lib/python2.7/site-packages/eventlet/hubs/hub.py:312(sleep_until)
                                                                                      2 0.000 0.000 /usr/lib/python2.7/site-packages/eventlet/hubs/hub.py:403(prepare_timers)
                                                                                      0 0.000 0.000 /usr/lib/python2.7/site-packages/eventlet/hubs/hub.py:438(fire_timers)
                                                                                      1 0.000 0.000 /usr/lib/python2.7/site-packages/eventlet/hubs/poll.py:76(wait)
                                                                                     30 0.000 0.000 /usr/lib/python2.7/site-packages/nova/network/model.py:470(__iter__)
                                                                                      0 0.000 0.000 /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:6013(_get_instance_disk_info)
                                                                                      0 0.000 0.000 /usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py:591(plug)
                                                                                      1 0.000 0.000 /usr/lib/python2.7/site-packages/oslo_serialization/jsonutils.py:206(loads)
                                                                                      1 0.000 0.000 /usr/lib64/python2.7/site-packages/libvirt.py:464(XMLDesc)
                                                                                      1 0.000 0.000 /usr/lib64/python2.7/site-packages/libvirt.py:1716(name)
                                                                                      2 0.000 0.000 {time.time}

[Henry Gessau (gessau)]

I am marking this as 'incomplete' until we find a way to do agent profiling that is compatible.

[Ramu Ramamurthy (ramu-ramamurthy)]

closing this as there is not much to be done about this for now,