Add option for health monitoring on provisioned load balancers

[Paul Goins]

A Kubernetes customer had several load balancers provisioned which got into a state where roughly 50% of connections were being dropped. Some of the backends were having problems, but since health monitoring wasn't enabled, requests were still being served to the bad backends until we manually enabled health monitoring while trying to root cause the problem with the backends.

Having an option to provision health monitoring at the time of load balancer provisioning would allow us to mitigate a situation like this to some degree.

[Drew Freiberger (afreiberger)]

To be clear, when Kubernetes creates a service endpoint, it calls to the openstack integrator to create a neutron/octavia loadbalancer to provide the public IP for the service and creates back-end pool members based on the currently running pods of the replicaset.

When the backend for neutron can't reach the network endpoints of some of the pods, there is a denial of service in the round-robin format for each connection that is assigned to one of the unreachable backends.

The kubernetes environment can't monitor/know about the network access issues in the underlying cloud topology, so it can't remove/add pods as the network is segmented, however, adding an openstack healthmonitor to the pool for the loadbalancer can help the back-end loadbalancer to check network connectivity to the various pods and mark down those which are unreachable.

The suggestion is that the integrator, when creating a roundrobin TCP loadbalancer should also add a healthmonitor to the loadbalancer to avoid this situation.

There may also need to be further integration done to report back the health of the backend loadbalancer as part of the service endpoint status in the k8s cluster.

[Drew Freiberger (afreiberger)]

actual monitoring/alerting of the status of these pool members manged by the healthmonitor would be related to lp#1853886

[Drew Freiberger (afreiberger)]

To add context of the impact of this bug, when a single kubernetes-master unit of 3 dies, it sets the nodes of roughly 1/3rd of all kubernetes workers to NotReady (and node monitor commands timeout) due to the loadbalancer not skipping the dead kubernetes-master unit.

This means that the lack of health monitoring in the kubernetes-master loadbalancer can cause significant reduction in cluster compute resource availability.

[Nobuto Murata (nobuto)]

In terms of Amphora in front of k8s-master, I thought it wasn't critical thanks to "option redispatch" in haproxy.conf inside the Amphora instance. In any case, health monitor for k8s-master is definitely good to have. I've tested it in this one fwiw:
https://bugs.launchpad.net/charm-openstack-integrator/+bug/1937171

[Drew Freiberger (afreiberger)]

FYI, to create a basic TCP service health monitor to validate network and service availability for any given loadbalancer pool:

openstack loadbalancer healthmonitor create --delay 5 --max-retries 4 --timeout 10 --type TCP <lbaas-pool-id>

[Drew Freiberger (afreiberger)]

@nobuto Thanks for the added details regarding "option redispatch". When I observed this, my CLI was doing pretty well with the API and getting consistent results.

It seems I've unearthed a different bug!

I see in my kubelet logs an attempt to directly talk to a single kubernetes-master unit's IP to query the API from the kubelet.

2022-02-01T15:17:39Z kubelet.daemon[3830984]: E0201 15:17:39.710823 3830984 controller.go:178] failed to update node lease, error: Put "https://192.168.200.123:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/juju-e06743-kubernetes-0?timeout=10s": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

So, ultimately, the failure mode I experienced seems to actually be related to which IP is provided to the kubernetes-worker over the relation. I'm going to gather additional information now.