sm services loss of redundancy alarms on install of storage system

[Maria Yousaf]

Brief Description
-----------------
On install of storage system, loss of redundancy alarms were observed for sm services.

Severity
--------
Critical

Steps to Reproduce
------------------
Install system

Expected Behavior
------------------
System should be alarm free at the end of install

Actual Behavior
----------------
The following alarms were seen:
+----------+--------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------+----------+----------------------------+
| Alarm ID | Reason Text | Entity ID | Severity | Time Stamp |
+----------+--------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------+----------+----------------------------+
| 400.002 | Service group controller-services loss of redundancy; expected 1 standby member but no standby members available | service_domain=controller.service_group=controller-services | major | 2018-10-11T05:26:25.835758 |
| 400.002 | Service group cloud-services loss of redundancy; expected 1 standby member but no standby members available | service_domain=controller.service_group=cloud-services | major | 2018-10-11T05:26:25.673766 |
| 400.002 | Service group vim-services loss of redundancy; expected 1 standby member but no standby members available | service_domain=controller.service_group=vim-services | major | 2018-10-11T05:26:25.512765 |
| 400.002 | Service group storage-services loss of redundancy; expected 2 active members but only 1 active member available | service_domain=controller.service_group=storage-services | major | 2018-10-11T05:26:22.911771 |
| 400.002 | Service group storage-monitoring-services loss of redundancy; expected 1 standby member but no standby members available | service_domain=controller.service_group=storage-monitoring-services | major | 2018-10-11T05:26:22.667809 |
| 400.002 | Service group patching-services loss of redundancy; expected 1 standby member but no standby members available | service_domain=controller.service_group=patching-services | major | 2018-10-11T05:02:06.811585 |
| 400.002 | Service group oam-services loss of redundancy; expected 1 standby member but no standby members available | service_domain=controller.service_group=oam-services | major | 2018-10-11T05:02:06.730590 |
| 400.002 | Service group directory-services loss of redundancy; expected 2 active members but only 1 active member available | service_domain=controller.service_group=directory-services | major | 2018-10-11T05:02:04.964594 |
| 400.002 | Service group web-services loss of redundancy; expected 2 active members but only 1 active member available | service_domain=controller.service_group=web-services | major | 2018-10-11T05:02:04.721579 |
+----------+--------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------+----------+----------------------------+

Reproducibility
---------------
Tried once.

System Configuration
--------------------
Storage system

Branch/Pull Time/Commit
-----------------------
master as of 2018-10-10_20-18-00

Timestamp/Logs
--------------
2018-10-11T05:26:25.512765

[Ghada Khalil (gkhalil)]

Targeting stx.2019.03 as this is an issue in stx master only, not in the r/2018.10 release branch

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/610990

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/610990
Committed: https://git.openstack.org/cgit/openstack/stx-ha/commit/?id=5f9731c47eb1ca5e5a970a9400464b22a7bacca3
Submitter: Zuul
Branch: master

commit 5f9731c47eb1ca5e5a970a9400464b22a7bacca3
Author: Bin Qian <email address hidden>
Date: Tue Oct 16 08:59:04 2018 -0400

    Fix service groups on controller-1 stuck in initial

    Initialize fail-pending timer id.
    Also deregister the timer when program exits.

    Closes-Bug: 1797567

    Change-Id: Ief278dfff1185a6acea718b683da11934a192161
    Signed-off-by: Bin Qian <email address hidden>