Comment 6 for bug 1838411

Here is a summary of the time taken to recover from an active controller reset (force reboot). Some notes:
- I did my testing on a 2+2 lab (IP_33-36 in Ottawa) with the 20190811T053000Z load and the stx-openstack application installed.
- Recovery time was measured using “openstack server list”, with a timeout of 15s.

Recovery times range from about 90s to 120s, with an average of about 105s. Here is a rough summary of where I think the time is going (times can be +/- 5s or so):

0s - force reboot
4s - SM begins enabling services
15s - all SM services enabled
25s - kube-controller-manager becomes leader (leader election time is 15s but requires etcd to be running)
40s - kube-controller-manager finishes recovering system state and starts timing for NodeNotReady
60s - kube-controller-manager declares that formerly active controller is NodeNotReady (20s node-monitor-grace-period) and removes the endpoints for pods running on that node

At this point (roughly 60s) the flock services and kubernetes components should be available and have an up-to-date view of the system state. However, it takes another 30-60s for the “openstack server list” command to start functioning again. I’m not 100% sure about this but I think it is related to the endpoints for some openstack pods not being available yet. I see two pods in particular that have issues:
- The osh-openstack-memcached-memcached pod is not replicated, so it takes time (30s or longer) for this pod to be re-created on the newly active controller. Some notes:
  - Without this pod, none of the openstack APIs are working (I confirmed this by removing the pod by setting the replicas to 0).
  - By default, kubernetes waits 30s to delete the pod and re-create it on the newly active controller (due to the “node.kubernetes.io/unreachable:NoExecute for 30s” toleration). I tried to set this toleration to 5s, but that didn’t improve things (not sure why).
  - I also tried to set the replicas to 2 for this pod and it also didn’t help. I’m not sure this pod supports more than 1 replica - see https://blueprints.launchpad.net/openstack-helm/+spec/memcached-resiliency for a proposed change to support memcached replication.
- The keystone-api pod often fails its liveness probe on the newly active controller, which causes the container to be restarted and the keystone-api endpoint to disappear for a significant period of time. I don’t know why this is happening.