check_node can block longer than nagios is willing to wait

[Adam Dyess]

when check_node command runs, it calls via subprocess

/snap/bin/kubectl --kubeconfig /var/lib/nagios/.kube/config get no -o=yaml

If this command blocks for 20 or 30 seconds, NRPE gives up before then resulting in an invalid check.

If i run the command locally (30s to fail):
$ sudo /snap/bin/kubectl --kubeconfig /var/lib/nagios/.kube/config get no -o=yaml
Unable to connect to the server: net/http: TLS handshake timeout

I can see there's an issue connecting to the master. Well that's nice -- but nagios didn't know this happened.

Maybe increase the expected timeout or update the check to catch this type of error

[George Kraft (cynerva)]

We should be able to repro this by deploying Charmed Kubernetes with Nagios, stopping the snap.kube-apiserver.daemon service on all masters, and then observing the kubernetes-worker status as reported by Nagios.

[Adam Dyess (addyess)]

Raised PR to address: https://github.com/charmed-kubernetes/charm-kubernetes-worker/pull/70

[George Kraft (cynerva)]

It looks like this went out with 1.19+ck1.