openvswitch-agent eating 100% cpu

[Ben Nemec]

I'm seeing this in my recent undercloud installs. For some reason the Neutron openvswitch-agent process is pegged at 100% cpu as soon as the undercloud is installed (so no overcloud deployed yet). I'll attach the log. I didn't notice anything obviously wrong, but maybe someone else will see an issue.

[Ben Nemec (bnemec)]

I see this in the ci logs as well, so it is almost certainly affecting our job runtimes and resource usage:

system_u:system_r:neutron_t:s0 neutron 4115 ***98.0*** 1.2 313944 99980 ? Rs 20:02 74:08 /usr/bin/python2 /usr/bin/neutron-openvswitch-agent ...

[Bogdan Dobrelya (bogdando)]

I can observe the same behavior in a containerized UC

[Sergii Golovatiuk (sgolovatiuk)]

Pike doesn't have such problems. I have just checked on my env.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/554539

[Ben Nemec (bnemec)]

I also did not see this on Pike.

However, I am seeing the same thing on the overcloud. It appears to be a problem with openvswitch-agent itself or our configuration of it, not something specific to the undercloud.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/554863

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/554869

[Bogdan Dobrelya (bogdando)]

At very least, I proposed related CI w/a to save some HW resources for the bug fix time being

[Harald Jensås (harald-jensas)]

This is probably neutron bug: https://bugs.launchpad.net/neutron/+bug/1750777

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/554539
Committed: https://git.openstack.org/cgit/openstack/paunch/commit/?id=4a4f43ac36c3edc2645c8fff5cf783415ea3f1cf
Submitter: Zuul
Branch: master

commit 4a4f43ac36c3edc2645c8fff5cf783415ea3f1cf
Author: Bogdan Dobrelya <email address hidden>
Date: Tue Mar 20 14:22:36 2018 +0100

    Allow to limit cgroup cpu shares

    Related-bug: #1755605

    Change-Id: I1f24866ea2ec01e3ef22571145efa25ab219bb3a
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[OpenStack Infra (hudson-openstack)]

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: master
Review: https://review.openstack.org/554863

[OpenStack Infra (hudson-openstack)]

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: master
Review: https://review.openstack.org/554869
Reason: Ping it if we might want this in the bug scope

[Ben Nemec (bnemec)]

I don't recall how or when this was fixed, but I don't see this behavior anymore.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.opendev.org/701064

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/701064
Committed: https://git.openstack.org/cgit/openstack/paunch/commit/?id=2ddf40d833f905353b9ca43a34e40f0118f0f898
Submitter: Zuul
Branch: stable/queens

commit 2ddf40d833f905353b9ca43a34e40f0118f0f898
Author: Bogdan Dobrelya <email address hidden>
Date: Tue Mar 20 14:22:36 2018 +0100

    Allow to limit cgroup cpu shares

    Related-bug: #1755605

    Change-Id: I1f24866ea2ec01e3ef22571145efa25ab219bb3a
    Signed-off-by: Bogdan Dobrelya <email address hidden>
    (cherry picked from commit 4a4f43ac36c3edc2645c8fff5cf783415ea3f1cf)