HA: control plane API performance can be degraded during a controller outage

[Damien Ciabrini]

In the HA control plane, many API services rely on memcached for caching ephemeral data that are often accessed. The API services access the memcached service directly, so they are configured with the list of all memcached servers (how they are accessing the servers is out of scope):

   # grep -m1 -B3 '^memcache' /var/lib/config-data/puppet-generated/nova/etc/nova/nova.conf
   # Memcache servers in the format of "host:port". (dogpile.cache.memcache and
   # oslo_cache.memcache_pool backends only) (list value)
   #memcache_servers=localhost:11211
   memcache_servers=172.17.1.115:11211,172.17.1.62:11211,172.17.1.60:11211

When a controller node is in outage, it can happen that the API service access a memcached server that is hosted on an unresponsive node, and the recovery from an unsuccessful memcache access will cost some time, degrading the API service's response time.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "James Slagle <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/773094
Reason: Abandoning this patch per the TripleO Patch Abandonment guidelines
(https://specs.openstack.org/openstack/tripleo-specs/specs/policy/patch-abandonment.html).
If you wish to have this restored and cannot do so yourself, please reach out
via #tripleo on OFTC or the OpenStack Dev mailing list.