There is a delay in API responses if one of Controllers is down.

[Alexander Rubtsov]

--- Environment ---
MOS 9.2 (build 606)

--- Description ---
There is a delay in API responses if one of Controllers is down.

--- Steps to reproduce ---
1) Deploy an HA environment
2) Measure response time of some CLI command:
http://paste.openstack.org/show/2gFAFNc6OsX1AcRvjgAq/
3) Shutdown a Controller node
4) Measure response time again:
http://paste.openstack.org/show/VeRD8iTrf6SFIlQaGfdl/

--- Actual result ---
There is a delay in responses

--- Expected result ---
Responses time take the same time as initial ones.

--- Notes ---
The main delays are related to VIP connections:
...
[05/23/18 07:35:39.789417450] INFO (connectionpool:242) Resetting dropped connection: 192.168.0.2
...
[05/23/18 07:35:43.857501946] INFO (connectionpool:207) Starting new HTTP connection (1): 192.168.0.2
...
(Full output is here: http://paste.openstack.org/show/8rxEE4gGAH4dErQOFnAI/)

[Alexander Rubtsov (arubtsov)]

sla2 for 9.0-updates

[Denis Meltsaykin (dmeltsaykin)]

This is known behavior of the memcache library used in almost any component through keystone middleware/client, unfortunately there is nothing we can do since it would take a major refactoring/development to introduce a new memcache sharding algorithm or get rid of sharding at all. This behavior is considered acceptable since it only happens in case of a failure of a controller, not in normal daily service. To avoid it you should run a re-deployment deleting a failed controller or adding a new one, which is more preferred. Then you will get a standard working configuration.

I'm going to set this report as Invalid.