[18.05] config-changed fails on rsyncing a nagios check due to a non-existing target dir (nrpe is not installed)

[Dmitrii Shcherbakov]

Single-unit rabbitmq-server, no nagios or nrpe in the model

App Version Status Scale Charm Store Rev OS Notes
rabbitmq-server 3.6.10 error 1 rabbitmq-server jujucharms 77 ubuntu

Unit Workload Agent Machine Public address Ports Message
rabbitmq-server/0* error idle 2/lxd/5 10.232.6.197 hook failed: "config-changed"

juju config rabbitmq-server | pastebinit
http://paste.ubuntu.com/p/vC7STqNFVc/

juju debug-log -i rabbitmq-server/0 --replay | pastebinit
http://paste.ubuntu.com/p/SHR26KrstZ/

unit-rabbitmq-server-0: 01:21:02 DEBUG unit.rabbitmq-server/0.config-changed os.path.join(NAGIOS_PLUGINS, 'check_rabbitmq_cluster.py'))
unit-rabbitmq-server-0: 01:21:02 DEBUG unit.rabbitmq-server/0.config-changed File "/var/lib/juju/agents/unit-rabbitmq-server-0/charm/hooks/charmhelpers/core/host.py", line 497, in rsync
unit-rabbitmq-server-0: 01:21:02 DEBUG unit.rabbitmq-server/0.config-changed return subprocess.check_output(cmd, stderr=subprocess.STDOUT).decode('UTF-8').strip()
unit-rabbitmq-server-0: 01:21:02 DEBUG unit.rabbitmq-server/0.config-changed File "/usr/lib/python2.7/subprocess.py", line 574, in check_output
unit-rabbitmq-server-0: 01:21:02 DEBUG unit.rabbitmq-server/0.config-changed raise CalledProcessError(retcode, cmd, output=output)
unit-rabbitmq-server-0: 01:21:02 DEBUG unit.rabbitmq-server/0.config-changed subprocess.CalledProcessError: Command '['/usr/bin/rsync', '-r', '--delete', '--executability', '/var/lib/juju/agents/unit-rabbitmq-server-0/charm/scripts/check_rabbitmq_cluster.py', '/usr/local/lib/nagios/plugins/check_rabbitmq_cluster.py']' returned non-zero exit status 3
unit-rabbitmq-server-0: 01:21:02 ERROR juju.worker.uniter.operation hook "config-changed" failed: exit status 1

ubuntu@juju-06074d-2-lxd-5:~$ stat /var/lib/juju/agents/unit-rabbitmq-server-0/charm/scripts/check_rabbitmq_cluster.py
  File: '/var/lib/juju/agents/unit-rabbitmq-server-0/charm/scripts/check_rabbitmq_cluster.py'
  Size: 3330 Blocks: 8 IO Block: 4096 regular file
Device: 801h/2049d Inode: 4983339 Links: 1
Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2018-08-09 00:40:35.373994726 +0000
Modify: 2018-08-09 00:40:35.373994726 +0000
Change: 2018-08-09 00:40:35.373994726 +0000
 Birth: -

ubuntu@juju-06074d-2-lxd-5:~$ stat /usr/local/lib/nagios/plugins/
stat: cannot stat '/usr/local/lib/nagios/plugins/': No such file or directory

ubuntu@juju-06074d-2-lxd-5:~$ /usr/bin/rsync -r --delete --executability /var/lib/juju/agents/unit-rabbitmq-server-0/charm/scripts/check_rabbitmq_cluster.py /usr/local/lib/nagios/plugins/check_rabbitmq_cluster.py
rsync: change_dir#3 "/usr/local/lib/nagios/plugins" failed: No such file or directory (2)
rsync error: errors selecting input/output files, dirs (code 3) at main.c(712) [Receiver=3.1.1]

ubuntu@juju-06074d-2-lxd-5:~$ dpkg -l | grep -i nagios ; echo $?
1

[Dmitrii Shcherbakov (dmitriis)]

This is quite problematic in conjunction with this behavior in Juju https://bugs.launchpad.net/juju/+bug/1786165

workaround:

# juju ssh <unit> sudo -i
root@juju-06074d-2-lxd-6:~# mkdir -p /usr/local/lib/nagios/plugins/
# juju resolved <unit>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/603813

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/18.08
Review: https://review.openstack.org/603814

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/603813
Committed: https://git.openstack.org/cgit/openstack/charm-rabbitmq-server/commit/?id=d5f28378d2974e353cee190b28ce859a1a063a8a
Submitter: Zuul
Branch: master

commit d5f28378d2974e353cee190b28ce859a1a063a8a
Author: Edward Hope-Morley <email address hidden>
Date: Wed Sep 19 15:11:35 2018 +0100

    Fix missing part of commit 4d4c998

    In that commit this piece of code was moved under the check
    for whether the nagios plugins dir exists but seems this piece
    was inadvertently not removed.

    Change-Id: I72d48438105edd8cb311d36253a592009a9c3706
    Closes-Bug: 1786164

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/603814
Committed: https://git.openstack.org/cgit/openstack/charm-rabbitmq-server/commit/?id=ea9e7f9db366b611cc9d385af336cc6f4a7c38c0
Submitter: Zuul
Branch: stable/18.08

commit ea9e7f9db366b611cc9d385af336cc6f4a7c38c0
Author: Edward Hope-Morley <email address hidden>
Date: Wed Sep 19 15:11:35 2018 +0100

    Fix missing part of commit 4d4c998

    In that commit this piece of code was moved under the check
    for whether the nagios plugins dir exists but seems this piece
    was inadvertently not removed.

    Change-Id: I72d48438105edd8cb311d36253a592009a9c3706
    Closes-Bug: 1786164
    (cherry picked from commit d5f28378d2974e353cee190b28ce859a1a063a8a)