config-changed IOError: [Errno 2] No such file or directory: '/var/lib/nagios/service-check-haproxy.txt' after recent update to check_status_file.py

[Ryan Beisner]

When resyncing charm-helpers into the Glance charm, the following hook failure exists (for at least the Trusty + Icehouse combo):

config-changed IOError: [Errno 2] No such file or directory: '/var/lib/nagios/service-check-haproxy.txt' after recent update to check_status_file.py

2017-03-03 22:22:19 INFO config-changed Setting up python-gi (3.12.0-1ubuntu1) ...
2017-03-03 22:22:19 INFO worker.uniter.jujuc server.go:172 running hook tool "juju-log" ["Check command not found: check_status_file.py"]
2017-03-03 22:22:19 DEBUG worker.uniter.jujuc server.go:173 hook context id "glance/0-config-changed-2954441461654224842"; dir "/var/lib/juju/agents/unit-glance-0/charm"
2017-03-03 22:22:19 INFO juju-log Check command not found: check_status_file.py
2017-03-03 22:22:19 INFO config-changed Traceback (most recent call last):
2017-03-03 22:22:19 INFO config-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/config-changed", line 632, in <module>
2017-03-03 22:22:19 INFO config-changed hooks.execute(sys.argv)
2017-03-03 22:22:19 INFO config-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/hookenv.py", line 731, in execute
2017-03-03 22:22:19 INFO config-changed self._hooks[hook_name]()
2017-03-03 22:22:19 INFO config-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1864, in wrapped_f
2017-03-03 22:22:19 INFO config-changed restart_functions)
2017-03-03 22:22:19 INFO config-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/host.py", line 655, in restart_on_change_helper
2017-03-03 22:22:19 INFO config-changed r = lambda_f()
2017-03-03 22:22:19 INFO config-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1863, in <lambda>
2017-03-03 22:22:19 INFO config-changed (lambda: f(*args, **kwargs)), restart_map, stopstart,
2017-03-03 22:22:19 INFO config-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/contrib/hardening/harden.py", line 79, in _harden_inner2
2017-03-03 22:22:19 INFO config-changed return f(*args, **kwargs)
2017-03-03 22:22:19 INFO config-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/config-changed", line 405, in config_changed
2017-03-03 22:22:19 INFO config-changed update_nrpe_config()
2017-03-03 22:22:19 INFO config-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/config-changed", line 593, in update_nrpe_config
2017-03-03 22:22:19 INFO config-changed nrpe.add_init_service_checks(nrpe_setup, services(), current_unit)
2017-03-03 22:22:19 INFO config-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/contrib/charmsupport/nrpe.py", line 387, in add_init_service_checks
2017-03-03 22:22:19 INFO config-changed f = open(checkpath, 'w')
2017-03-03 22:22:19 INFO config-changed IOError: [Errno 2] No such file or directory: '/var/lib/nagios/service-check-haproxy.txt'
2017-03-03 22:22:19 INFO juju.worker.uniter.context context.go:579 handling reboot

Logs and artifacts available for viewing at:

https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_amulet_full/openstack/charm-glance/441237/2/725/index.html

[Ryan Beisner (1chb1n)]

See related change:
https://code.launchpad.net/~paulgear/charm-helpers/nrpe-service-immediate-check/+merge/300682

[Ryan Beisner (1chb1n)]

It seems that C-H commit changed the default behavior with immediate_check=True.

[Felipe Reyes (freyes)]

I think here charm-helpers here is not the one at fault, because /var/lib/nagios will be created once nagios-nrpe-server is installed and here glance is calling update_nrpe_config() withouth checking if it's related to nrpe or if nagios-nrpe-server is installed.

glance should be verify it before it attempts to write the configuration.

[Paul Gear (paulgear)]

@freyes If you find that this is the fault of my recent charm-helpers change, or need any assistance in tracking it down, feel free to drag me in...

[Felipe Reyes (freyes)]

@paulgear, thanks, for the offer. I came up with a simple patch, if you could take a look would be great.

The other I was thinking is to make sure charms (in this case glance) don't call to add_init_service_checks() before nagios-nrpe-server is installed, but I'm not sure if it's a good approach considering that we may have to end up patching multiple charms due to the change in the behavior.

[Paul Gear (paulgear)]

@freyes Added a question to your MP; but +1 to your point that patching multiple charms is going to be harder than making add_init_service_checks() handle it. I wonder why these charms were doing things like that in the first place, though, without having the relation to the nrpe charm already established.

[Ryan Beisner (1chb1n)]

FYI The keystone fails in the same way after a c-h resync. Artifacts and logs @:

https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_amulet_full/openstack/charm-keystone/442714/1/740/index.html

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/441237
Committed: https://git.openstack.org/cgit/openstack/charm-glance/commit/?id=896ea18db03aef92f0d4bfaf9885c6fdcde38c5a
Submitter: Jenkins
Branch: master

commit 896ea18db03aef92f0d4bfaf9885c6fdcde38c5a
Author: Felipe Reyes <email address hidden>
Date: Fri Mar 3 14:15:34 2017 -0300

    charm-helpers sync to pickup changes to CephContext

    CephContext now is capable of dealing with a space separated list of IPs
    in the ceph-public-address field of the relation with ceph. This allows
    ceph-proxy to pass a list of monitor hosts instead of a single one.

    Change-Id: I83abc2c151636aea0b5dffc32f2e3c91acb7558b
    Partial-Bug: 1642430
    Closes-Bug: 1670223

[Liam Young (gnuoy)]

Marking as fix released for keystone as the charmhelper fix was pulled into the keystone charm for the last charm release.