Default ulimit value for neutron containers (DHCP, L3 and OVS agents) limits a number of SDN entities in production environment

[Slawek Kaplonski]

Bug originally reported by Alex Stupnikov:

Description of problem:

TripleO sets ulimits for different containers since RHOSP 13 [1].
Default value for neutron services and nova-compute is 1024. This
default doesn't work well for nova-compute with huge number of attached
disks, so I have reported bug [2].

Today customer informed me that neutron services are ALSO AFFECTED:
he faced issues with DHCP agents after creating 250 tenant networks
(quite low number) and was able to solve them after increasing default
ulimit value by defining DockerNeutronDHCPAgentUlimit.

I have asked neutron developers to provide recommendations for defaults for the following parameters and was told to report the bug:

DockerNeutronDHCPAgentUlimit
DockerNeutronL3AgentUlimit
DockerOpenvswitchUlimit

[1] https://review.openstack.org/#/c/560991/
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1693667

[Alex Schultz (alex-schultz)]

So for the record we lowered it to 1024 because rootwrap didn't like the larger default value. See Bug 1760471 So we to figure out a happy medium between 131072 and 1024

[Slawek Kaplonski (slaweq)]

Hi Alex,

Yes, it seems that we lowered it too much when fixing rootwrap issue.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/651137
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=5d3fe630db336b54284a05720f5ef0d3fc8bd729
Submitter: Zuul
Branch: master

commit 5d3fe630db336b54284a05720f5ef0d3fc8bd729
Author: Slawek Kaplonski <email address hidden>
Date: Tue Apr 9 10:26:27 2019 +0200

    Increase default ulimit values for Neutron agents containers

    For containers where Neutron DHCP, L3 and OVS agents are running
    there is limit for number of opened files set.
    This limit was configured by default to 1024 which looks that is
    quite low and can cause problems even with relatively small number
    of networks or routers handled by agent.

    This patch increase default ulimit values for those containers to be
    16x higher than it was before. Such value should allow to handle much
    more routers or networks and shouldn't be too high for any system.

    Change-Id: Ibc8fa0c1ccd4f60ec7c876623a7262003197f67d
    Closes-bug: #1824020

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/651709

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/651710

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/651709
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=178773f92b0baf28f9cec93e5e7527c510a10ac7
Submitter: Zuul
Branch: stable/rocky

commit 178773f92b0baf28f9cec93e5e7527c510a10ac7
Author: Slawek Kaplonski <email address hidden>
Date: Tue Apr 9 10:26:27 2019 +0200

    Increase default ulimit values for Neutron agents containers

    For containers where Neutron DHCP, L3 and OVS agents are running
    there is limit for number of opened files set.
    This limit was configured by default to 1024 which looks that is
    quite low and can cause problems even with relatively small number
    of networks or routers handled by agent.

    This patch increase default ulimit values for those containers to be
    16x higher than it was before. Such value should allow to handle much
    more routers or networks and shouldn't be too high for any system.

    Conflicts:
        deployment/neutron/neutron-ovs-agent-container-puppet.yaml

    Change-Id: Ibc8fa0c1ccd4f60ec7c876623a7262003197f67d
    Closes-bug: #1824020
    (cherry picked from commit 5d3fe630db336b54284a05720f5ef0d3fc8bd729)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/651710
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=bc30fb036947473c53515ff96e800d8e4e3468da
Submitter: Zuul
Branch: stable/queens

commit bc30fb036947473c53515ff96e800d8e4e3468da
Author: Slawek Kaplonski <email address hidden>
Date: Tue Apr 9 10:26:27 2019 +0200

    Increase default ulimit values for Neutron agents containers

    For containers where Neutron DHCP, L3 and OVS agents are running
    there is limit for number of opened files set.
    This limit was configured by default to 1024 which looks that is
    quite low and can cause problems even with relatively small number
    of networks or routers handled by agent.

    This patch increase default ulimit values for those containers to be
    16x higher than it was before. Such value should allow to handle much
    more routers or networks and shouldn't be too high for any system.

    Conflicts:
        deployment/neutron/neutron-ovs-agent-container-puppet.yaml

    Change-Id: Ibc8fa0c1ccd4f60ec7c876623a7262003197f67d
    Closes-bug: #1824020
    (cherry picked from commit 5d3fe630db336b54284a05720f5ef0d3fc8bd729)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 10.5.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 9.4.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 8.4.0 release.