oslo.cache doesn't close connections to memcached

[Thomas Goirand]

oslo.cache creates a connection pool to memcached, though the __del__ hasn't been implemented. As a consequence, not-closed connections are left in a CLOSE_WAIT state, which expires after some time (which time depends on the server configuration, as this is customizable with sysctl).

Under a heavy load, such as for example our gnocchi API in production, where we have a huge load every 5 minutes, there's so many CLOSE_WAIT opened sockets that we don't even have any source port available: all 65k ports are left in a CLOSE_WAIT state.

The solution is simple (as per the opened PR): close sockets when the connectionPool is deleted.

[Takashi Kajinami (kajinamit)]

A fix proposed to master https://review.opendev.org/c/openstack/oslo.cache/+/900158

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/oslo.cache/+/900158
Committed: https://opendev.org/openstack/oslo.cache/commit/bae5561d4d3ded0a7c22f4a3e90b2a8e535a5384
Submitter: "Zuul (22348)"
Branch: master

commit bae5561d4d3ded0a7c22f4a3e90b2a8e535a5384
Author: Thomas Goirand <email address hidden>
Date: Mon Nov 6 10:55:45 2023 +0100

    Clean-up memcache connection sockets

    The ConnectionPool currently opens sockets, but never closes them. As a
    result, any client using oslo.cache memcache pool leaves sockets in a
    CLOSE_WAIT state, with the source port not being re-usable.

    In our production system, were we have A LOT of activity, this creates
    a storm of non-reusable ports: all source ports are in use, making the
    node unuseable for other things.

    This patch adds a __del__ destructor closing inactive connections,
    fixing the issue.

    Closes-Bug: #2043121
    Change-Id: I09d632346c76d1aff7c534f0d040162d1985f548

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/oslo.cache 3.7.0 release.