check_haproxy_servers and check_haproxy_queue have blank command defs causing "NRPE: Command x not defined" errors

[Drew Freiberger]

On the most recent release of charms, 18.11 on a Xenial/Queens cloud, I'm noticing that I have all units of openstack-dashboard, keystone, and nova-cloud-compute showing errors in nagios.

NRPE: Command 'check_<random check name>' not defined

On each unit, I'm seeing checks unrelated to check_haproxy_servers and check_haproxy_queue also erroring as 'not defined', such as check_memcached or check_vhost or check_apache2, as well.

When investigating, I see that the error in sysctl is showing:

Nov 21 18:19:26 juju-45c58f-31-lxd-0 nagios-nrpe-server[118208]: * Starting nagios-nrpe nagios-nrpe
Nov 21 18:19:26 juju-45c58f-31-lxd-0 nrpe[118212]: No variable value specified in config file '/etc/nagios/nrpe.d/check_haproxy_servers.cfg' - Line 5
Nov 21 18:19:26 juju-45c58f-31-lxd-0 nrpe[118212]: Continuing with errors...
Nov 21 18:19:26 juju-45c58f-31-lxd-0 nrpe[118213]: Starting up daemon
Nov 21 18:19:26 juju-45c58f-31-lxd-0 nagios-nrpe-server[118208]: ...done.

$ cat /etc/nagios/nrpe.d/check_haproxy_servers.cfg
# check haproxy_servers
# The following header was added automatically by juju
# Modifying it will affect nagios monitoring and alerting
# servicegroups: juju
command[check_haproxy_servers]=

$ cat /etc/nagios/nrpe.d/check_haproxy_queue.cfg
# check haproxy_queue
# The following header was added automatically by juju
# Modifying it will affect nagios monitoring and alerting
# servicegroups: juju
command[check_haproxy_queue]=

The parsing of config files by the nrpe daemon stops after detecting this error, leaving other defined checks to fail.

[Drew Freiberger (afreiberger)]

In fact, the shell scripts check_haproxy.sh and check_haproxy_queue_depth.sh are not found in /usr/lib/nagios/plugins, which makes me think something changed in charmhelpers to stop landing those files upon nrpe.add-check calls.

[Drew Freiberger (afreiberger)]

I've at least been able to track down that the command is blank in the .cfg file because we're receiving "Check command not found: check_haproxy_queue_depth.sh" and "
Check command not found: check_haproxy.sh".

This comes from charmhelpers.contrib.openstack.charmsupport.nrpe.Check._locate_cmd which is searching for the existence of these files to configure the command in the nrpe .cfg files.

Something is missing in charmhelpers included in 18.11 charms that used to install these bits, I feel.

[Drew Freiberger (afreiberger)]

This is the root issue:

https://github.com/juju/charm-helpers/issues/246