collectd alarm clear request failure leads to stuck alarm

[Eric MacDonald]

A Starling-X collectd alarm notification handler FM (fault manager) call to clear an alarm can lead to a stuck alarm if that FM request fails, say due to a concurrent swact operation.

The alarm will remain stuck until there is another same alarm assertion and then successful clear.

Severity:
---------
Major

Steps to Reproduce
------------------
Not easy. Cause a collectd monitored alarm condition and then cause that condition to clear where the next audit and clear attempt is concurrent with a swact or FM process restart.

Expected Behavior
------------------
Alarm clear is retried by the collectd alarm handler on next audit interval.

Actual Behavior
----------------
No subsequent clear is retried until the next assertion, negation and then next clear.

Reproducibility
---------------
Intermittent/Rare - see steps to reproduce for explanation for reproducibility understanding.

System Configuration
--------------------
AIO DX.

Branch/Pull Time/Commit
-----------------------
BUILD_DATE="2018-09-19 20:19:35 -0400"

Timestamp/Logs
--------------
From /var/log/daemon.log on controller-1

2018-09-20T13:05:10.838 controller-1 collectd[167073]: info WARNING:root:fm_python_extension: Failed to connect to FM manager
2018-09-20T13:05:10.838 controller-1 collectd[167073]: info alarm notifier 100.101:host=controller-1 clear_fault failed
2018-09-20T13:05:10.838 controller-1 collectd[167073]: info alarm notifier cpu no warnings
2018-09-20T13:05:10.838 controller-1 collectd[167073]: info alarm notifier clear alarm 100.101:okay cpu:host=controller-1 thld:90.00 value:69.22

[Ghada Khalil (gkhalil)]

There is already a launchpad reporting this issue:
https://bugs.launchpad.net/starlingx/+bug/1793314

[Ghada Khalil (gkhalil)]

Marking this as a duplicate

[Ghada Khalil (gkhalil)]

Duplicate of https://bugs.launchpad.net/starlingx/+bug/1793314
Marking as Fix Released; same as the duplicate bug report. Fix was merged on 2018-09-20