swift-storage should tune conntrack

[James Troup]

I just got 'conntrack nearly full' alerts for a couple of nodes on a reasonable sized Swift cluster (~15 nodes, but with ½B+ objects). I did a quick check and of the 63K connections being tracked, 99.95% of them were accounted for by swift (ports 600{0,1,2} and 873). Because the charm puts a firewall on the hosts connection track will always be on, so it seems that the charm should tune the relevant conntrack sysctl settings like other charms (e.g. nova-compute) do.

(There is a sysctl juju config setting in the charm but it's empty by default.)

[James Page (james-page)]

Settings from nova-compute:

      { net.ipv4.neigh.default.gc_thresh1 : 128,
        net.ipv4.neigh.default.gc_thresh2 : 28672,
        net.ipv4.neigh.default.gc_thresh3 : 32768,
        net.ipv6.neigh.default.gc_thresh1 : 128,
        net.ipv6.neigh.default.gc_thresh2 : 28672,
        net.ipv6.neigh.default.gc_thresh3 : 32768,
        net.nf_conntrack_max : 1000000,
        net.netfilter.nf_conntrack_buckets : 204800,
        net.netfilter.nf_conntrack_max : 1000000 }

[James Page (james-page)]

Marking triaged as this is a well qualified bug with a clear path forwards.

[James Page (james-page)]

https://review.opendev.org/742602

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/742602

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-swift-storage/+/742602
Committed: https://opendev.org/openstack/charm-swift-storage/commit/d9a8025ac3d8fa4743970e20a5634cd18680d2e9
Submitter: "Zuul (22348)"
Branch: master

commit d9a8025ac3d8fa4743970e20a5634cd18680d2e9
Author: James Page <email address hidden>
Date: Thu Jul 23 08:03:01 2020 +0100

    conntrack: set sane sysctl defaults

    swift-storage is often deployed alongside nova-compute where it
    inherits some sane defaults for sysctl values, specifically around
    conntrack configuration.

    If its deployed standalone it does not recieve the same tuning,
    but its very much applicable as access between swift-storage nodes
    for rsync traffic is firewall limited so makes a high demand
    on conntrack on the server.

    Lift and shift the defaults from the nova-compute charm.

    Change-Id: Iedd27a51ff93fd1670a418e1434c94875fe21643
    Closes-Bug: 1879121