Neutron SRIOV agent loop iteration exceeding polling interval, causing high CPU usage in neutron-rootwrap-daemon

[Rodolfo Alonso]

According to [1], the problem is due to the excessive maximum number of resources granted to the container. In [2] this number was limited to 1024 for neutron-dhcp, neutron-l3 and neutron-ovs-agent containers, but we are experiencing the same problem in the neutron-sriov-agent container.

The following code was tested (execute "ip link show") using rootwrap and privsep [3]:
- ulimit unlimited: 800ms with rootwrap
- ulimit 4096: 30ms

[1] https://bugs.launchpad.net/oslo.rootwrap/+bug/1760471
[2] https://review.openstack.org/#/c/559268/
[3] http://paste.openstack.org/show/748949/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/650406

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/650416

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/650418

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/650406
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=b01a57491e8cebd4d6a4117e00740edba59f9b61
Submitter: Zuul
Branch: master

commit b01a57491e8cebd4d6a4117e00740edba59f9b61
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Fri Apr 5 18:00:49 2019 +0000

    Set ulimit 16384 for Neutron SR-IOV container

    Same a in [1], rootwrap takes too much time if ulimit is the
    default value. In the related bug, the execution time drops
    from 800ms to 30ms if ulimit is set to 16384.

    [1] https://review.openstack.org/#/c/559268/

    Change-Id: I18de5b733066853a8c057596e85299048cee8604
    Closes-Bug: #1823385

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/650416
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=12543005b11cd8f2873cea48123ed190d5c8327b
Submitter: Zuul
Branch: stable/rocky

commit 12543005b11cd8f2873cea48123ed190d5c8327b
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Fri Apr 5 18:00:49 2019 +0000

    Set ulimit 16384 for Neutron SR-IOV container

    Same a in [1], rootwrap takes too much time if ulimit is the
    default value. In the related bug, the execution time drops
    from 800ms to 30ms if ulimit is set to 16384.

    [1] https://review.openstack.org/#/c/559268/

    Change-Id: I18de5b733066853a8c057596e85299048cee8604
    Closes-Bug: #1823385

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/650418
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=35b4bf6e039c132f32cd107753938fecc79ec340
Submitter: Zuul
Branch: stable/queens

commit 35b4bf6e039c132f32cd107753938fecc79ec340
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Fri Apr 5 18:00:49 2019 +0000

    Set ulimit 16384 for Neutron SR-IOV container

    Same a in [1], rootwrap takes too much time if ulimit is the
    default value. In the related bug, the execution time drops
    from 800ms to 30ms if ulimit is set to 16384.

    [1] https://review.openstack.org/#/c/559268/

    Change-Id: I18de5b733066853a8c057596e85299048cee8604
    Closes-Bug: #1823385
    (cherry picked from commit 12543005b11cd8f2873cea48123ed190d5c8327b)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 10.5.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 9.4.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 8.4.0 release.