"node" NRPE check fails due to permissions on /var/lib/nagios

[James Hebden]

kubernetes v1.17.2
kubernetes-worker charm from cs:kubernetes-worker-634
Ubuntu bionic deployed on an OpenStack controller

The "node" NRPE check fails to run kubectl, as it runs under NRPE, which runs as the 'nagios' system user - by default, the '/var/lib/nagios' directory is owned by root, which means the /usr/lib/nagios/plugins/check_k8s_worker.py check script can't create the /var/lib/nagios/snap directory, which it uses during the kubectl run to check API registration status.

[Zachary Zehring (zzehring)]

I ran into this issue as well on a production environment. Notably, this did not occur in all of our nodes. 2 out of 6 actually had the correct permissions set for the /usr/lib/nagios directory.

A workaround for the remaining nodes was just running the following:

sudo chown nagios:nagios /var/lib/nagios/

[George Kraft (cynerva)]

Sounds like it might be a race condition. I'm guessing that /var/lib/nagios is supposed to be owned by nagios:nagios, but the kubernetes-worker call to create_kubeconfig[1] is causing it to get created as root:root if it does not exist by then.

[1]: https://github.com/charmed-kubernetes/charm-kubernetes-worker/blob/a23db7e33c8203022685bbfa43185f2b6c5f90b3/reactive/kubernetes_worker.py#L1039

[Chris Johnston (cjohnston)]

Is this related to LP#1866382?

[George Kraft (cynerva)]

Yes, LP#1866382 appears to be the same issue. I'm marking this one as a duplicate. Thanks!