OVS L2 agent polling is too cpu intensive

[Maru Newby]

On a Devstack-deployed, single-node install, the ovs l2 agent is using an order of magnitude more cpu than any other service. On a nested-virt VM running on a 2.5GHz host, with no VM's provisioned:

 - WIth L2 agent running, 10-100% cpu usage recorded, averaging ~40%
 - With L2 agent stopped, 0-10% cpu usage recorded

Casual inspection with something like top or htop will show the excessive cpu usage, but won't give a good indication of the culprit. Enabling the CUTIME and CSTIME stats is necessary to highlight the problem, as all the time taken by the agent is in subprocess-invoked polling whose execution time is not directly included in the parent's cpu usage.

[Maru Newby (maru)]

The current polling-based checks for ovs bridge changes would ideally be replaced by something event-based.

[Édouard Thuleau (ethuleau)]

We encounter the same problem with the Grizzly release and OVS plugin.
I set up a node which handle l2 agent, l3 agent and DHCP agent.
I doesn't modify intervals 'agent_down_time' (5sec) and 'report_interval' (4sec).

When the l2 agent has about 25 interfaces to handle, the pooling-base check take more time than the interval 'report_interval':

2013-06-03 15:44:21 WARNING [quantum.openstack.common.loopingcall] task run outlasted interval by 0.80157 sec

And the Quantum server think that the agent is down because the report state takes more than interval 'agent_down_time'.

It's possible to change the intervals value to correct the problem but when the number of interfaces increases these intervals time have a chance to be reach.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/34762

[Julian Sternberg (jules-i)]

any solution for this yet? the patch got abandoned?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/43256

[Maru Newby (maru)]

My intention is to provide a new event-based approach that will not be turned on by default so as to minimize the potential for breakage leading up to the Havana release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/43256
Committed: http://github.com/openstack/neutron/commit/1f9b4e77d93b165a0aeeaae0de610389debd5198
Submitter: Jenkins
Branch: master

commit 1f9b4e77d93b165a0aeeaae0de610389debd5198
Author: Maru Newby <email address hidden>
Date: Thu Aug 22 07:57:00 2013 +0000

    Minimize ovs l2 agent calls to get_vif_port_set()

    The ovs l2 agent was previously calling get_vif_port_set() on the
    integration bridge once per rpc_loop() iteration and then again in
    the periodic _report_state() call that returns the current device
    count to the neutron service. Since get_vif_port_set() is an
    expensive call (relying on shell commands) and since there
    is minimal risk associated with reporting stats that are a few
    seconds old, this patch caches the device count for reuse by
    _report_state().

    Partial-Bug: 1177973

    Change-Id: Ice73384ed1ba1e97120028cd0a9bff94a62a41a4

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/45676

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/45677

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/45678

[Salvatore Orlando (salvatore-orlando)]

I am the one who moved this target to RC1, but I now wonder if the new 4 patches which have been pushed are still targeting Havana, or are meant to be merged in Icehouse.

I am not entirely sure these patches fulfill the criteria needed for acceptance in RC phase; comments welcome.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/45676
Committed: http://github.com/openstack/neutron/commit/acf0209b28e21eed60158967fab77468eb195e7c
Submitter: Jenkins
Branch: master

commit acf0209b28e21eed60158967fab77468eb195e7c
Author: Maru Newby <email address hidden>
Date: Mon Sep 9 01:29:54 2013 -0700

    Add support for managing async processes

    Interacting with a long-running asynchronous process requires the
    use of non-blocking io. This change adds a helper class that can
    launch a long-running process and read stdout and stderr in a
    non-blocking fashion via eventlet.

    This functionality is intended to support monitoring ovsdb via
    a long-running and root-privileged invocation of ovsdb-client.

    The complexity of the system interaction in this patch suggested
    the addition of a functional test that validated actual behaviour.
    The test was added under the neutron/tests/functional path which
    is now included in the testr search path.

    Partial-Bug: #1177973

    Change-Id: I9969e556acecf7a9e77d873371cc2ec2647be011

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/45677
Committed: http://github.com/openstack/neutron/commit/010bd1f392e67a6fcd276593b8c79acfe41d1cc7
Submitter: Jenkins
Branch: master

commit 010bd1f392e67a6fcd276593b8c79acfe41d1cc7
Author: Maru Newby <email address hidden>
Date: Mon Sep 9 09:58:12 2013 +0000

    Improve ovs_lib bridge management

    Add the ability to add and remove bridges, check for bridge
    existence, and lookup the bridge associated with a port.

    This change is in support of functional testing for an ovsdb
    monitor.

    Partial-Bug: #1177973

    Change-Id: I419923d8d77983997cd347fcf063b0bc367c0bbc

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/45678
Committed: http://github.com/openstack/neutron/commit/cb0df591a9508e863ad5d5d71190eca349dc551f
Submitter: Jenkins
Branch: master

commit cb0df591a9508e863ad5d5d71190eca349dc551f
Author: Maru Newby <email address hidden>
Date: Mon Sep 9 10:06:49 2013 +0000

    Add the option to minimize ovs l2 polling

    This change adds the ability to monitor the local ovsdb for
    interface changes so that the l2 agent can avoid unnecessary
    polling. Minimal changes are made to the agent so the risk
    of breakage should be low. Future efforts to make the agent
    entirely event-based may be able to use OvsdbMonitor as a
    starting point.

    By default polling minimization is not done, and can only be
    enabled by setting 'minimize_polling = True' in the ovs
    section of the l2 agent's config file.

    Closes-Bug: #1177973

    Change-Id: I26c035b48a74df2148696869c5a9affae5ab3d27

[Peter Feiner (pete5)]

I wonder why this isn't considered havana backport potential anymore? Using minimze_polling has a significant effect on performance, as the bug report describes. Moreover, when too much polling happens, the openvswitch agent starts to thrash, making the host useless for starting new instances. See http://lists.openstack.org/pipermail/openstack-dev/2013-December/021323.html.

[Salvatore Orlando (salvatore-orlando)]

I said I would support the backport, but I forgot this patch adds a dependency (psutil I think)

I hope we can find a way to work around this issue.

For the comments on the OVS agent, we've observed improvements testified by significant reduction of "timeouts" failures on the gate since enabling it. However, the agent under load can still be quite a mess. For more info see: https://bugs.launchpad.net/neutron/+bug/1253993

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/65808

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/65809

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/65810

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/65808
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2e9f5a7a3156ad91c901099f7577179f0c588a9c
Submitter: Jenkins
Branch: stable/havana

commit 2e9f5a7a3156ad91c901099f7577179f0c588a9c
Author: Maru Newby <email address hidden>
Date: Mon Sep 9 01:29:54 2013 -0700

    Add support for managing async processes

    Interacting with a long-running asynchronous process requires the
    use of non-blocking io. This change adds a helper class that can
    launch a long-running process and read stdout and stderr in a
    non-blocking fashion via eventlet.

    This functionality is intended to support monitoring ovsdb via
    a long-running and root-privileged invocation of ovsdb-client.

    The complexity of the system interaction in this patch suggested
    the addition of a functional test that validated actual behaviour.
    The test was added under the neutron/tests/functional path which
    is now included in the testr search path.

    Partial-Bug: #1177973

    Change-Id: I9969e556acecf7a9e77d873371cc2ec2647be011
    (cherry picked from commit acf0209b28e21eed60158967fab77468eb195e7c)

[Alan Pevec (apevec)]

Only first patch in the series[1] was merged to stable/havana

[1] https://review.openstack.org/#/q/project:openstack/neutron+branch:stable/havana+topic:bug/1243867,n,z

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/65809
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6dd647ae73a404724f11b422f5fc1bb04e48aecf
Submitter: Jenkins
Branch: stable/havana

commit 6dd647ae73a404724f11b422f5fc1bb04e48aecf
Author: Maru Newby <email address hidden>
Date: Mon Sep 9 09:58:12 2013 +0000

    Improve ovs_lib bridge management

    Add the ability to add and remove bridges, check for bridge
    existence, and lookup the bridge associated with a port.

    This change is in support of functional testing for an ovsdb
    monitor.

    Partial-Bug: #1177973

    Change-Id: I419923d8d77983997cd347fcf063b0bc367c0bbc
    (cherry picked from commit 010bd1f392e67a6fcd276593b8c79acfe41d1cc7)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/65810
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=39ef5187b5d0bd9484f8e5773e90ba112fcf06e1
Submitter: Jenkins
Branch: stable/havana

commit 39ef5187b5d0bd9484f8e5773e90ba112fcf06e1
Author: Maru Newby <email address hidden>
Date: Mon Sep 9 10:06:49 2013 +0000

    Add the option to minimize ovs l2 polling

    This change adds the ability to monitor the local ovsdb for
    interface changes so that the l2 agent can avoid unnecessary
    polling. Minimal changes are made to the agent so the risk
    of breakage should be low. Future efforts to make the agent
    entirely event-based may be able to use OvsdbMonitor as a
    starting point.

    By default polling minimization is not done, and can only be
    enabled by setting 'minimize_polling = True' in the ovs
    section of the l2 agent's config file.

    Closes-Bug: #1177973

    Change-Id: I26c035b48a74df2148696869c5a9affae5ab3d27
    (cherry picked from commit cb0df591a9508e863ad5d5d71190eca349dc551f)