[doc] Controller removed with Fuel when LMA is active results in Alarms

[Bruce Basil Mathews]

From Justin Stone (<email address hidden>) CitrixOne:

Fuel 7.0 updated to the latest MU. LMA/Stacklight patched to the latest version (.8)... Removing controller(s) and adding back using Fuel causes Grafana to alert on Neutron and Nova as it still thinks the old controllers are there but now down as they are a new node-id. I found a couple things about hiding the nodes in grafana but wasn’t sure if you had some better info on the inside. The Controllers were removed and added through the fuel 7.0 ui.

[Simon Pasquier (simon-pasquier)]

Thanks for your report. I've moved the bug to the lma-toolchain project since this is where we track the LMA bugs.

[Simon Pasquier (simon-pasquier)]

We probably need to document it better but there is a flaw in Fuel that doesn't decommission the services associated to the removed node.

Here are the manual operations that you need to run to remove the alerts:
1. Check that the Nova, Neutron & Cinder services for the removed controller are still registered.
$ neutron agent-list
$ nova service-list
$ cinder service-list
2. Delete or remove the obsolete services
$ neutron agent-delete ...
$ cinder service-disable ...
$ nova service-delete ...
3. Restart the collector service on the controller nodes
$ crm resource restart lma_collector

Tell us if the procedure works for you.

[Justin Stone (stoney66)]

Hi Simon,

The procedure worked, now grafana is back to OK status for neutron/nova (3 controllers up, 0 down). The old controllers still show in the drop down menu on far left side though but the graph data is blank when selecting them.

[Simon Pasquier (simon-pasquier)]

Thanks for the feedback Justin! To hide items in the drop-down menu, please refer to the latest documentation [1].

[1] http://fuel-plugin-influxdb-grafana.readthedocs.org/en/latest/user.html#hiding-nodes-from-dashboards

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/344211
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-lma-collector/commit/?id=a2972647ddce62bed9e52a06d3030d5c30569742
Submitter: Jenkins
Branch: master

commit a2972647ddce62bed9e52a06d3030d5c30569742
Author: Patrick Petit <email address hidden>
Date: Tue Jul 19 14:28:35 2016 +0200

    StackLight 0.10.0 documentation updates

    Closes-Bug: #1569027

    Change-Id: Ia8c2915d19a175a341288344b412e95ded79ec27

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/0.10
Review: https://review.openstack.org/345391

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/345391
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-lma-collector/commit/?id=ec6f51c3012a3a5cfed94cd4f7f855109cf7fd99
Submitter: Jenkins
Branch: stable/0.10

commit ec6f51c3012a3a5cfed94cd4f7f855109cf7fd99
Author: Patrick Petit <email address hidden>
Date: Tue Jul 19 14:28:35 2016 +0200

    StackLight 0.10.0 documentation updates

    Closes-Bug: #1569027

    Change-Id: Ia8c2915d19a175a341288344b412e95ded79ec27
    (cherry picked from commit a2972647ddce62bed9e52a06d3030d5c30569742)