Perf regression in SEA network provisioning

[Jeremy Arnold]

Deploy performance testing of current Newton code is showing a significant regression compared to Mitaka in nova's plug_vifs, which relies on the neutron sea_agent provisioning. With moderate scale (5 hosts, 400 VMs, 10 concurrent deploys), the median plug_vifs time in my tests has increased 3-4x.

My analysis suggests that the sea_agent is now generating more get_device_details / update_device_up cycles due to the recent removal of the PVIDLooper, because the CNAEventHandler is now triggering more provisioning operations which used to be coalesced by PVIDLooper. These additional port status changes drive notifications across neutron and nova on both the host and controller, and this extra activity is getting in the way of the actual deploy operations.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/385592

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/378000
Committed: https://git.openstack.org/cgit/openstack/networking-powervm/commit/?id=c35eb18de15434a50efe0dd2690136515d2b2d1b
Submitter: Jenkins
Branch: master

commit c35eb18de15434a50efe0dd2690136515d2b2d1b
Author: Eric Fried <email address hidden>
Date: Wed Oct 5 16:14:07 2016 -0500

    Refactor, consolidate, and clean up agents

    This change set represents a major refactor of the neutron SEA and
    SR-IOV agents.

    - Proper abstract properties/methods allow much code to be consolidated
      into the BasePVMNeutronAgent from the individual subclasses.
    - Conversely, CNA-specific code, such as helpers for the
      sea_agent-specific heal_and_optimize loop, is moved to the sea_agent
      module.
    - PVMRpcCallbacks methods are folded into the agent base class itself.
      The agent is now the thing that's registered for those callbacks.
    - The network_delete callback is gone (it wasn't being used).
    - The port_update callback is a no-op by default. In the SR-IOV agent,
      it's refreshing the physical port label:physloc mappings back to
      neutron.
    - The vnic_required_vfs and vnic_vf_capacity conf options are moved to
      sriov_agent.
    - There is no longer a short timer in the rpc_loop; everything is
      event-driven. Thus, the polling_interval conf option is removed.
    - The rpc_loop is common to both agents. It only runs
      heal_and_optimize, on the configurable interval (default 30min).
    - ProvisionRequest is moved into its own module, prov_req.py. It is now
      a factory class capable of producing ProvisionRequest instances for:
      - Custom VIF Events from the nova-powervm VIF driver;
      - pypowervm Wrappers of VIF types (CNA, VNIC);
    - The VIFEventHandler (formerly CNAEventHandler) is now driven by VIF
      events produced by the nova-powervm VIF driver. This entails a
      significant processing/performance/load improvement over the previous
      code, which acted on all LPAR events.
    - Utility methods (utils.py) are renamed and genericized to handle all
      VIF types (CNA, VNIC) rather than just CNAs.
    - Obsolete utility methods and their unit tests are removed.
    - Unit tests overhauled and extended.

    Closes-Bug: #1632400

    Change-Id: Ie8a13da0b0cdc29d350ed91d96ff2384430c2676

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/385592
Committed: https://git.openstack.org/cgit/openstack/networking-powervm/commit/?id=e1a287a56cfea4e6fb9ca29458ef436bbe27fc72
Submitter: Jenkins
Branch: stable/newton

commit e1a287a56cfea4e6fb9ca29458ef436bbe27fc72
Author: Eric Fried <email address hidden>
Date: Wed Oct 5 16:14:07 2016 -0500

    Refactor, consolidate, and clean up agents

    This change set represents a major refactor of the neutron SEA and
    SR-IOV agents.

    - Proper abstract properties/methods allow much code to be consolidated
      into the BasePVMNeutronAgent from the individual subclasses.
    - Conversely, CNA-specific code, such as helpers for the
      sea_agent-specific heal_and_optimize loop, is moved to the sea_agent
      module.
    - PVMRpcCallbacks methods are folded into the agent base class itself.
      The agent is now the thing that's registered for those callbacks.
    - The network_delete callback is gone (it wasn't being used).
    - The port_update callback is a no-op by default. In the SR-IOV agent,
      it's refreshing the physical port label:physloc mappings back to
      neutron.
    - The vnic_required_vfs and vnic_vf_capacity conf options are moved to
      sriov_agent.
    - There is no longer a short timer in the rpc_loop; everything is
      event-driven. Thus, the polling_interval conf option is removed.
    - The rpc_loop is common to both agents. It only runs
      heal_and_optimize, on the configurable interval (default 30min).
    - ProvisionRequest is moved into its own module, prov_req.py. It is now
      a factory class capable of producing ProvisionRequest instances for:
      - Custom VIF Events from the nova-powervm VIF driver;
      - pypowervm Wrappers of VIF types (CNA, VNIC);
    - The VIFEventHandler (formerly CNAEventHandler) is now driven by VIF
      events produced by the nova-powervm VIF driver. This entails a
      significant processing/performance/load improvement over the previous
      code, which acted on all LPAR events.
    - Utility methods (utils.py) are renamed and genericized to handle all
      VIF types (CNA, VNIC) rather than just CNAs.
    - Obsolete utility methods and their unit tests are removed.
    - Unit tests overhauled and extended.

    Closes-Bug: #1632400

    Change-Id: Ie8a13da0b0cdc29d350ed91d96ff2384430c2676