The cron job update-lma-configuration lma_infrastructure_alerting removes configuration files when we apply the nagios.pp

[guillaume thouvenin]

Something very strange happens on MOS 8 environment after an upgrade from SL 0.9.0 to 0.10.1. For the upgrade the following document has been used: https://docs.google.com/document/d/1BE74LOkAlK5Hy1RJAtrSsfrmw8EVwR56_ucaCenN3z4

After the upgrade the nagios3 was not able to start. The configuration files prefixed by "lma_" in the /etc/nagios3 were not present on all three SL nodes. To reconfigure nagios we run several times the following command:

# puppet apply --debug --modulepath=/etc/fuel/plugins/lma_infrastructure_alerting-1.0/puppet/modules:/etc/puppet/modules /etc/fuel/plugins/lma_infrastructure_alerting-1.0/puppet/manifests/nagios.pp

Each time the configuration files were created but right after they were deleted.

We modified the file /etc/fuel/plugins/lma_infrastructure_alerting-1.0/puppet/modules/lma_infra_alerting/manifests/nagios.pp to be sure that the cron is not installed and it worked.

If we run the cron job manually on the node we got the following error:

root@lma-2:/var/tmp# /usr/bin/flock -n /tmp/lma.lock -c "/usr/local/bin/update-lma-configuration lma_infrastructure_alerting"
Error: Could not run: Could not find file /etc/fuel/plugins/lma_infrastructure_alerting-0.10/puppet
/etc/fuel/plugins/lma_infrastructure_alerting-0.9/puppet/manifests/nagios.pp
root@lma-2:/var/tmp# ls /etc/nagios3/
apache2.conf cgi.cfg commands.cfg conf.d htpasswd.users nagios.cfg resource.cfg stylesheets
root@lma-2:/var/tmp# ls /etc/nagios3/conf.d/
cmd_notify-service-by-email-with-long-service-output.cfg generic-host_nagios2.cfg localhost_nagios2.cfg.puppet-bak
contacts_nagios2.cfg generic-service_nagios2.cfg services_nagios2.cfg.puppet-bak
extinfo_nagios2.cfg.puppet-bak hostgroups_nagios2.cfg.puppet-bak timeperiods_nagios2.cfg
root@lma-2:/var/tmp#

And lma_ configuration files have been removed.

[guillaume thouvenin (guillaume-thouvenin)]

There are two problems:

1) The script is setting PLUGIN_PUPPET_DIR=$(ls -d /etc/fuel/plugins/"$PLUGIN_NAME"*/puppet). It works when we have one version of the plugin, but if we have two versions then the puppet apply failed to apply because PLUGIN_PUPPET_DIR="lma_infrastructure_alerting-0.9 lma_infrastructure_alerting-0.10". It is the error we are seeing in the description.
2) And as the script failed, the file LAST_CHECK and LAST_CHECK_NODES are never updated. So the cron job is running over and over....

We need to fix the script to check what version of the plugin is running.

[guillaume thouvenin (guillaume-thouvenin)]

I'm currently testing but if two versions of the plugin are available we should hit the bug. So I change the severity to high.

[guillaume thouvenin (guillaume-thouvenin)]

In fact the problem occurs only after an upgrade. In other cases only the version that you are installing will be copied on the nodes. After an upgrade, both versions are available and you hit the bug.

[guillaume thouvenin (guillaume-thouvenin)]

To reproduce the bug:

- You can simulate the result of an upgrade by copying the content of the /etc/fuel/plugins/lma_infrastructure_alerting. For example if you have 0.10 installed:

cp -r /etc/fuel/plugins/lma_infrastructure_alerting-0.10/ /etc/fuel/plugins/lma_infrastructure_alerting-0.9

Then modify /var/cache/lma_last_astute_yaml.md5sum to force the execution of the cron job.

After one minute you will see in the /etc/nagios3/conf.d directory that all "lma_*" files disappeared.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/369211

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/369211
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-lma-infrastructure-alerting/commit/?id=bccb10031c7ad120bbdd939067a0af3097c57d84
Submitter: Jenkins
Branch: master

commit bccb10031c7ad120bbdd939067a0af3097c57d84
Author: Guillaume Thouvenin <email address hidden>
Date: Tue Sep 13 10:06:33 2016 +0200

    Modify the cron job to use specific version of the plugin

    This patch modifies the cron job used to update the nagios configuration
    to solve a bug that occurs when several version of the plugin are
    available. This happens after an upgrade procedure.

    Change-Id: I41831e43707e7ca5e88fd0a329508ad4813d26bb
    Closes-Bug: #1622628

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/0.10
Review: https://review.openstack.org/373959

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/373959
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-lma-infrastructure-alerting/commit/?id=68ecd65fc21b4ac1677a589206b41f3a59469f7a
Submitter: Jenkins
Branch: stable/0.10

commit 68ecd65fc21b4ac1677a589206b41f3a59469f7a
Author: Guillaume Thouvenin <email address hidden>
Date: Tue Sep 13 10:06:33 2016 +0200

    Modify the cron job to use specific version of the plugin

    This patch modifies the cron job used to update the nagios configuration
    to solve a bug that occurs when several version of the plugin are
    available. This happens after an upgrade procedure.

    Change-Id: I41831e43707e7ca5e88fd0a329508ad4813d26bb
    Closes-Bug: #1622628
    (cherry picked from commit bccb10031c7ad120bbdd939067a0af3097c57d84)