[RFE] Improve profiling of port binding and vif plugging

[Bence Romsics]

As discussed on the 2019-May PTG in Denver we want to measure then improve the performance of Neutron's most important operation that is port binding.

As we're working with OSProfiler reports we are realizing the report is incomplete. We could turn on tracing in other components and subcomponents by further propagating trace information.

We heavily build on some previous work:

* https://bugs.launchpad.net/neutron/+bug/1335640 [RFE] Neutron support for OSprofiler
* https://review.opendev.org/615350 Integrate rally with osprofiler

A few patches were already merged before opening this RFE:

* https://review.opendev.org/662804 Run nova's VM boot rally scenario in the neutron gate
* https://review.opendev.org/665614 Allow VM booting rally scenarios to time out

We already see the need for a few changes:

* New rally scenario to measure port binding
* Profiling coverage for vif plugging

This work is also driven by the discoveries made while interpreting profiler reports so I expect further changes here and there.

[YAMAMOTO Takashi (yamamoto)]

it isn't clear to me what's the purpose of this RFE.
has this already agreed on in Denver?
do you just want to track the progress of the effort within this RFE?

[Bence Romsics (bence-romsics)]

Hi Yamamoto,

Yes, we discussed most of this in Denver and I just would like to track all the small litte changes.

Bence

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/670008

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/670009

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/662786
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=5e0a5d41c7e4d16ebc670d7629d72a67299a07cb
Submitter: Zuul
Branch: master

commit 5e0a5d41c7e4d16ebc670d7629d72a67299a07cb
Author: Bence Romsics <email address hidden>
Date: Fri May 31 13:35:26 2019 +0200

    Rally task definition for port binding scenario

    Change-Id: I112e477f95958c6235dbc42da06ebc45b236e8f6
    Depends-On: https://review.opendev.org/662781
    Partial-Bug: #1833674

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/666532
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=45e59e3a1838c1a2069c5cb759060fedb81727c6
Submitter: Zuul
Branch: master

commit 45e59e3a1838c1a2069c5cb759060fedb81727c6
Author: Bence Romsics <email address hidden>
Date: Thu Jun 20 12:10:21 2019 +0200

    Propagate profiler info into BatchNotifier threads

    While working on improving the osprofiler report in neutron I'm finding
    various places where the profiling can be extended by propagating the
    profiler info further then before.

    This change starts to profile BatchNotifier threads (e.g. various
    callbacks to nova) when the code starting the thread was already
    profiled (it's not at the moment, but that will be another change).

    In this change we use osprofiler.profiler.clean(). Since that has became
    part of the public interface of osprofiler in version 2.3.0, we bump
    osprofiler version to 2.3.0 both in lower-constraints and requirements.

    Change-Id: Ibd08e097b6f8457c50f8ba9e4a63b96e7e3182bc
    Partial-Bug: #1833674

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/670008
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0299746c72b1a5f3d0c2ca1921f8febb1fb977b3
Submitter: Zuul
Branch: master

commit 0299746c72b1a5f3d0c2ca1921f8febb1fb977b3
Author: Bence Romsics <email address hidden>
Date: Tue Jun 25 15:55:36 2019 +0200

    ovs-agent: Trace remote methods only

    Enabling osprofiler tracing on class OVSNeutronAgent decorates all of
    its methods. Doing this as label "rpc" for locally called methods is
    quite misleading.

    More importantly later we want to enable tracing triggered by a vif
    plug. For that profiler.init() must be called sometime in
    OVSNeutronAgent.process_ports_events(). If process_ports_events() is
    decorated for tracing and we call profiler.init() inside then we'll have
    the end of the method call traced, but not its beginning. The unmatching
    end-trace-event makes osprofiler raise.

    Change-Id: I06b8643c75a30b5a5f6f78d75af9937180da645b
    Partial-Bug: #1833674

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/671780

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/671780
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e8caea24513e6a6b4b461ea5efb262f02075bfbc
Submitter: Zuul
Branch: master

commit e8caea24513e6a6b4b461ea5efb262f02075bfbc
Author: Bence Romsics <email address hidden>
Date: Fri Jul 19 13:46:22 2019 +0200

    Trace receipt of update_device_* rpc messages in neutron-server

    Generate an osprofiler trace_point every time neutron-server receives
    an update_device_* rpc message with trace_info in its context. We assume
    this marks the time when ovs-agent finished processing a port.

    Change-Id: I554bc45ad44931691029e0066d5b8483522a7dd2
    Partial-Bug: #1833674

[OpenStack Infra (hudson-openstack)]

Change abandoned by Bence Romsics (<email address hidden>) on branch: master
Review: https://review.opendev.org/666610
Reason: As discussed on the Ussuri PTG in Shanghai this is way more complicated than using cProfile so I'm abandoning this in favor of I5c847a255b4b973add7fce5a38a2594912c491ac.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.opendev.org/670009
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[Brian Haley (brian-haley)]

This seems to be complete, will close bug. Please re-open if I'm wrong.