Ceph-mon had failed the active controller, causing swact rollback.
| 2019-09-05T17:55:12.677 |        565 | service-scn          | hw-mon                           | enabling                         | enabled-active                   | enable success
| 2019-09-05T17:56:02.854 |        566 | service-scn          | ceph-mon                         | enabling                         | disabling                        | enable failed
| 2019-09-05T17:58:03.580 |        567 | service-scn          | ceph-mon                         | disabling                        | disabling-failed                 | disable failed
| 2019-09-05T17:58:03.581 |        568 | service-group-scn    | controller-services              | go-active                        | go-active-failed                 | ceph-mon(disabling, failed)
| 2019-09-05T17:58:04.477 |        569 | service-scn          | ceph-mon                         | disabling-failed                 | enabling-failed                  | enabled-active state requested
| 2019-09-05T17:58:07.793 |        570 | service-group-scn    | vim-services                     | active                           | disabling                        |
| 2019-09-05T17:58:07.794 |        571 | service-group-scn    | cloud-services                   | active                           | disabling                        |
| 2019-09-05T17:58:07.795 |        572 | service-group-scn    | controller-services              | go-active-failed                 | disabling-failed                 | ceph-mon(enabling, failed)
| 2019-09-05T17:58:07.796 |        573 | service-group-scn    | oam-services                     | active                           | disabling                        |

2019-09-05T17:55:02.000 controller-0 sm: debug time[7529.198] log<1426> INFO: sm[107278]: sm_service_enable.c(461): Started enable action (1008359) for service (ceph-mon).
2019-09-05T17:55:02.000 controller-0 sm: debug time[7529.198] log<1427> INFO: sm[107278]: sm_service_fsm.c(1032): Service (ceph-mon) received event (enable-throttle) was in the enabling-throttle state and is now in the enabling state.
2019-09-05T17:55:32.000 controller-0 sm: debug time[7559.216] log<1572> INFO: sm[107278]: sm_service_enable.c(245): Action (enable) timeout with result (failed), state (unknown), status (unknown), and condition (unknown) for service (ceph-mon), reason_text=, exit_code=-65534.
2019-09-05T17:55:32.000 controller-0 sm: debug time[7559.216] log<1573> INFO: sm[107278]: sm_service_action.c(345): Aborting service (ceph-mon) with kill signal, pid=1008359.
2019-09-05T17:55:32.000 controller-0 sm: debug time[7559.216] log<1574> INFO: sm[107278]: sm_service_enable.c(154): Max retires not met for action (enable) of service (ceph-mon), attempts=1.
2019-09-05T17:55:32.000 controller-0 sm: debug time[7559.216] log<1575> INFO: sm[107278]: sm_service_enable.c(461): Started enable action (1013901) for service (ceph-mon).
2019-09-05T17:56:02.000 controller-0 sm: debug time[7589.223] log<1577> INFO: sm[107278]: sm_service_enable.c(245): Action (enable) timeout with result (failed), state (unknown), status (unknown), and condition (unknown) for service (ceph-mon), reason_text=, exit_code=-65534.
2019-09-05T17:56:02.000 controller-0 sm: debug time[7589.223] log<1578> INFO: sm[107278]: sm_service_action.c(345): Aborting service (ceph-mon) with kill signal, pid=1013901.
2019-09-05T17:56:02.000 controller-0 sm: debug time[7589.223] log<1579> INFO: sm[107278]: sm_service_enable.c(149): Max retires met for action (enable) of service (ceph-mon), attempts=2.
2019-09-05T17:56:02.000 controller-0 sm: debug time[7589.224] log<1580> INFO: sm[107278]: sm_service_disable.c(380): Started disable action (1017517) for service (ceph-mon), flag (0).
2019-09-05T17:56:02.000 controller-0 sm: debug time[7589.224] log<1581> INFO: sm[107278]: sm_service_fsm.c(1032): Service (ceph-mon) received event (enable-failed) was in the enabling state and is now in the disabling state.
2019-09-05T17:56:32.000 controller-0 sm: debug time[7619.242] log<1584> INFO: sm[107278]: sm_service_disable.c(163): Action (disable) timeout with result (failed), state (unknown), status (unknown), and condition (unknown) for service (ceph-mon), reason_text=, exit_code=-65534.
2019-09-05T17:56:32.000 controller-0 sm: debug time[7619.242] log<1585> INFO: sm[107278]: sm_service_action.c(345): Aborting service (ceph-mon) with kill signal, pid=1017517.
2019-09-05T17:56:32.000 controller-0 sm: debug time[7619.242] log<1586> INFO: sm[107278]: sm_service_disable.c(69): Max retires met for action (disable) of service (ceph-mon), attempts=1.
2019-09-05T17:56:32.000 controller-0 sm: debug time[7619.243] log<1587> INFO: sm[107278]: sm_service_disable.c(380): Started disable action (1022130) for service (ceph-mon), flag (0).
2019-09-05T17:57:02.000 controller-0 sm: debug time[7649.265] log<1589> INFO: sm[107278]: sm_service_disable.c(163): Action (disable) timeout with result (failed), state (unknown), status (unknown), and condition (unknown) for service (ceph-mon), reason_text=, exit_code=-65534.
2019-09-05T17:57:02.000 controller-0 sm: debug time[7649.265] log<1590> INFO: sm[107278]: sm_service_action.c(345): Aborting service (ceph-mon) with kill signal, pid=1022130.
2019-09-05T17:57:02.000 controller-0 sm: debug time[7649.265] log<1591> INFO: sm[107278]: sm_service_disable.c(69): Max retires met for action (disable) of service (ceph-mon), attempts=1.
2019-09-05T17:57:03.000 controller-0 sm: debug time[7649.566] log<1593> INFO: sm[107278]: sm_service_disable.c(380): Started disable action (1025565) for service (ceph-mon), flag (0).
2019-09-05T17:57:33.000 controller-0 sm: debug time[7679.566] log<1594> INFO: sm[107278]: sm_service_disable.c(163): Action (disable) timeout with result (failed), state (unknown), status (unknown), and condition (unknown) for service (ceph-mon), reason_text=, exit_code=-65534.
2019-09-05T17:57:33.000 controller-0 sm: debug time[7679.566] log<1595> INFO: sm[107278]: sm_service_action.c(345): Aborting service (ceph-mon) with kill signal, pid=1025565.
2019-09-05T17:57:33.000 controller-0 sm: debug time[7679.566] log<1596> INFO: sm[107278]: sm_service_disable.c(69): Max retires met for action (disable) of service (ceph-mon), attempts=1.
2019-09-05T17:57:33.000 controller-0 sm: debug time[7679.940] log<1598> INFO: sm[107278]: sm_service_disable.c(380): Started disable action (1029204) for service (ceph-mon), flag (0).
2019-09-05T17:58:03.000 controller-0 sm: debug time[7709.951] log<1599> INFO: sm[107278]: sm_service_disable.c(163): Action (disable) timeout with result (failed), state (unknown), status (unknown), and condition (unknown) for service (ceph-mon), reason_text=, exit_code=-65534.
2019-09-05T17:58:03.000 controller-0 sm: debug time[7709.951] log<1600> INFO: sm[107278]: sm_service_action.c(345): Aborting service (ceph-mon) with kill signal, pid=1029204.
2019-09-05T17:58:03.000 controller-0 sm: debug time[7709.951] log<1601> INFO: sm[107278]: sm_service_disable.c(69): Max retires met for action (disable) of service (ceph-mon), attempts=1.
2019-09-05T17:58:03.000 controller-0 sm: debug time[7709.951] log<1602> INFO: sm[107278]: sm_service_disabling_state.c(259): Service (ceph-mon) is failed and has reached max action failures (4).
2019-09-05T17:58:03.000 controller-0 sm: debug time[7709.951] log<1603> INFO: sm[107278]: sm_service_fsm.c(1032): Service (ceph-mon) received event (disable-failed) was in the disabling state and is now in the disabling-failed state, condition=action-failure.
2019-09-05T17:58:03.000 controller-0 sm: debug time[7709.951] log<1604> INFO: sm[107278]: sm_service_group_audit.c(434): Service group (controller-services) member (ceph-mon) has mapped status (failed), overall_status=failed.
2019-09-05T17:58:03.000 controller-0 sm: debug time[7709.951] log<1605> INFO: sm[107278]: sm_service_group_audit.c(444): Service group (controller-services) member (ceph-mon) has mapped condition (action-failure), overall_condition=action-failure.
2019-09-05T17:58:04.000 controller-0 sm: debug time[7710.583] log<1612> INFO: sm[107278]: sm_service_api.c(300): Recovery of service (ceph-mon) requested.
2019-09-05T17:58:04.000 controller-0 sm: debug time[7710.847] log<1613> INFO: sm[107278]: sm_service_engine.c(347): Fail counts and status for service (ceph-mon) cleared in disabling state.
2019-09-05T17:58:04.000 controller-0 sm: debug time[7710.848] log<1614> INFO: sm[107278]: sm_service_enable.c(461): Started enable action (1032745) for service (ceph-mon).
2019-09-05T17:58:04.000 controller-0 sm: debug time[7710.848] log<1615> INFO: sm[107278]: sm_service_fsm.c(1032): Service (ceph-mon) received event (enable-throttle) was in the disabling-failed state and is now in the enabling-failed state.
2019-09-05T17:58:07.000 controller-0 sm: debug time[7714.365] log<1643> INFO: sm[107278]: sm_service_enable.c(477): Aborting action (enable) of service (ceph-mon).
2019-09-05T17:58:07.000 controller-0 sm: debug time[7714.365] log<1644> INFO: sm[107278]: sm_service_action.c(345): Aborting service (ceph-mon) with kill signal, pid=1032745.
2019-09-05T17:58:07.000 controller-0 sm: debug time[7714.366] log<1645> INFO: sm[107278]: sm_service_disable.c(380): Started disable action (1034053) for service (ceph-mon), flag (0).
2019-09-05T17:58:07.000 controller-0 sm: debug time[7714.366] log<1646> INFO: sm[107278]: sm_service_fsm.c(1032): Service (ceph-mon) received event (disable) was in the enabling-failed state and is now in the disabling-failed state.