Comment 0 for bug 1892852

We have recently installer an OpenStack cluster in Train release and we have noticed an unexpected behavior when Neutron contacts memcached upon some specific API requests. In fact, in our environment lbaas is not configured in Neutron (I don't know if this still possible in the current Neutron version), but we have deployed a monitoring service based on Prometheus openstack-exporter which, by default, checks the accessibility of the lbaas neutron part of the API.
Each time the check have the neutron server generates a 404, as this part of the API is not available. An example can be found at http://paste.openstack.org/show/797113/

The actual issue happens at the socket management level. Each time the check is performed, the socket between the Neutron server and memcached is not released as expected. This leads to a continuous increment of the established sockets between Neutron an Memcached until all the available sockets are exhausted and the cluster authentication locks.
Removing the checking on the lbaas service from Prometheus openstack-exporter allows to avoid the issue but we think that an issue like this (if confirmed) have quite important security concerns as it can lead quite easily to a DoS (wanted or not).

For the record, We have tried to explicitly configure the memcached socket monitoring options in Neutron but it looks like that they are not applied under the current conditions :
memcache_pool_socket_timeout = 3
memcache_pool_unused_timeout = 60

Here under is the environment setup:

Deployer : OpenStack Ansible 20.1.2

Neutron version: neutron 15.1.1.dev2

Neutron config file http://paste.openstack.org/show/797117/

Memcached config file http://paste.openstack.org/show/797118/

Prometheus OpenStack Exporter : https://github.com/openstack-exporter/openstack-exporter

Prometheus OpenStack Exporter config : http://paste.openstack.org/show/797119/

Prometheus command WITH lbaas checking : =/usr/local/bin/openstack-exporter \
    --os-client-config=/etc/openstack-exporter/openstack-exporter.yml \
    --web.telemetry-path="/metrics" \
    --web.listen-address="0.0.0.0:9180" \
    --prefix="openstack" \
    --endpoint-type="public"

Prometheus command WITHOUT lbaas checking:
/usr/local/bin/openstack-exporter \
    --os-client-config=/etc/openstack-exporter/openstack-exporter.yml \
    --web.telemetry-path="/metrics" \
    --web.listen-address="0.0.0.0:9180" \
    --prefix="openstack" \
    --endpoint-type="public" \
    --disable-metric neutron-loadbalancers \
    --disable-metric neutron-loadbalancers_not_active \
    --disable-service.object-store \
    --disable-service.load-balancer \

I hope the provided information is enough to reproduce the issue.

Cheers
Sergio