StarlingX

  1. Bug #1842966
  2. Comment #2

Comment 2 for bug 1842966

Revision history for this message
Bin Qian (bqian20) wrote : Re: unexpected swact occurred after host-swact completed

Ceph-mon had failed the active controller, causing swact rollback.
| 2019-09-05T17:55:12.677 | 565 | service-scn | hw-mon | enabling | enabled-active | enable success
| 2019-09-05T17:56:02.854 | 566 | service-scn | ceph-mon | enabling | disabling | enable failed
| 2019-09-05T17:58:03.580 | 567 | service-scn | ceph-mon | disabling | disabling-failed | disable failed
| 2019-09-05T17:58:03.581 | 568 | service-group-scn | controller-services | go-active | go-active-failed | ceph-mon(disabling, failed)
| 2019-09-05T17:58:04.477 | 569 | service-scn | ceph-mon | disabling-failed | enabling-failed | enabled-active state requested
| 2019-09-05T17:58:07.793 | 570 | service-group-scn | vim-services | active | disabling |
| 2019-09-05T17:58:07.794 | 571 | service-group-scn | cloud-services | active | disabling |
| 2019-09-05T17:58:07.795 | 572 | service-group-scn | controller-services | go-active-failed | disabling-failed | ceph-mon(enabling, failed)
| 2019-09-05T17:58:07.796 | 573 | service-group-scn | oam-services | active | disabling |

2019-09-05T17:55:02.000 controller-0 sm: debug time[7529.198] log<1426> INFO: sm[107278]: sm_service_enable.c(461): Started enable action (1008359) for service (ceph-mon).
2019-09-05T17:55:02.000 controller-0 sm: debug time[7529.198] log<1427> INFO: sm[107278]: sm_service_fsm.c(1032): Service (ceph-mon) received event (enable-throttle) was in the enabling-throttle state and is now in the enabling state.
2019-09-05T17:55:32.000 controller-0 sm: debug time[7559.216] log<1572> INFO: sm[107278]: sm_service_enable.c(245): Action (enable) timeout with result (failed), state (unknown), status (unknown), and condition (unknown) for service (ceph-mon), reason_text=, exit_code=-65534.
2019-09-05T17:55:32.000 controller-0 sm: debug time[7559.216] log<1573> INFO: sm[107278]: sm_service_action.c(345): Aborting service (ceph-mon) with kill signal, pid=1008359.
2019-09-05T17:55:32.000 controller-0 sm: debug time[7559.216] log<1574> INFO: sm[107278]: sm_service_enable.c(154): Max retires not met for action (enable) of service (ceph-mon), attempts=1.
2019-09-05T17:55:32.000 controller-0 sm: debug time[7559.216] log<1575> INFO: sm[107278]: sm_service_enable.c(461): Started enable action (1013901) for service (ceph-mon).
2019-09-05T17:56:02.000 controller-0 sm: debug time[7589.223] log<1577> INFO: sm[107278]: sm_service_enable.c(245): Action (enable) timeout with result (failed), state (unknown), status (unknown), and condition (unknown) for service (ceph-mon), reason_text=, exit_code=-65534.
2019-09-05T17:56:02.000 controller-0 sm: debug time[7589.223] log<1578> INFO: sm[107278]: sm_service_action.c(345): Aborting service (ceph-mon) with kill signal, pid=1013901.
2019-09-05T17:56:02.000 controller-0 sm: debug time[7589.223] log<1579> INFO: sm[107278]: sm_service_enable.c(149): Max retires met for action (enable) of service (ceph-mon), attempts=2.
2019-09-05T17:56:02.000 controller-0 sm: debug time[7589.224] log<1580> INFO: sm[107278]: sm_service_disable.c(380): Started disable action (1017517) for service (ceph-mon), flag (0).
2019-09-05T17:56:02.000 controller-0 sm: debug time[7589.224] log<1581> INFO: sm[107278]: sm_service_fsm.c(1032): Service (ceph-mon) received event (enable-failed) was in the enabling state and is now in the disabling state.
2019-09-05T17:56:32.000 controller-0 sm: debug time[7619.242] log<1584> INFO: sm[107278]: sm_service_disable.c(163): Action (disable) timeout with result (failed), state (unknown), status (unknown), and condition (unknown) for service (ceph-mon), reason_text=, exit_code=-65534.
2019-09-05T17:56:32.000 controller-0 sm: debug time[7619.242] log<1585> INFO: sm[107278]: sm_service_action.c(345): Aborting service (ceph-mon) with kill signal, pid=1017517.
2019-09-05T17:56:32.000 controller-0 sm: debug time[7619.242] log<1586> INFO: sm[107278]: sm_service_disable.c(69): Max retires met for action (disable) of service (ceph-mon), attempts=1.
2019-09-05T17:56:32.000 controller-0 sm: debug time[7619.243] log<1587> INFO: sm[107278]: sm_service_disable.c(380): Started disable action (1022130) for service (ceph-mon), flag (0).
2019-09-05T17:57:02.000 controller-0 sm: debug time[7649.265] log<1589> INFO: sm[107278]: sm_service_disable.c(163): Action (disable) timeout with result (failed), state (unknown), status (unknown), and condition (unknown) for service (ceph-mon), reason_text=, exit_code=-65534.
2019-09-05T17:57:02.000 controller-0 sm: debug time[7649.265] log<1590> INFO: sm[107278]: sm_service_action.c(345): Aborting service (ceph-mon) with kill signal, pid=1022130.
2019-09-05T17:57:02.000 controller-0 sm: debug time[7649.265] log<1591> INFO: sm[107278]: sm_service_disable.c(69): Max retires met for action (disable) of service (ceph-mon), attempts=1.
2019-09-05T17:57:03.000 controller-0 sm: debug time[7649.566] log<1593> INFO: sm[107278]: sm_service_disable.c(380): Started disable action (1025565) for service (ceph-mon), flag (0).
2019-09-05T17:57:33.000 controller-0 sm: debug time[7679.566] log<1594> INFO: sm[107278]: sm_service_disable.c(163): Action (disable) timeout with result (failed), state (unknown), status (unknown), and condition (unknown) for service (ceph-mon), reason_text=, exit_code=-65534.
2019-09-05T17:57:33.000 controller-0 sm: debug time[7679.566] log<1595> INFO: sm[107278]: sm_service_action.c(345): Aborting service (ceph-mon) with kill signal, pid=1025565.
2019-09-05T17:57:33.000 controller-0 sm: debug time[7679.566] log<1596> INFO: sm[107278]: sm_service_disable.c(69): Max retires met for action (disable) of service (ceph-mon), attempts=1.
2019-09-05T17:57:33.000 controller-0 sm: debug time[7679.940] log<1598> INFO: sm[107278]: sm_service_disable.c(380): Started disable action (1029204) for service (ceph-mon), flag (0).
2019-09-05T17:58:03.000 controller-0 sm: debug time[7709.951] log<1599> INFO: sm[107278]: sm_service_disable.c(163): Action (disable) timeout with result (failed), state (unknown), status (unknown), and condition (unknown) for service (ceph-mon), reason_text=, exit_code=-65534.
2019-09-05T17:58:03.000 controller-0 sm: debug time[7709.951] log<1600> INFO: sm[107278]: sm_service_action.c(345): Aborting service (ceph-mon) with kill signal, pid=1029204.
2019-09-05T17:58:03.000 controller-0 sm: debug time[7709.951] log<1601> INFO: sm[107278]: sm_service_disable.c(69): Max retires met for action (disable) of service (ceph-mon), attempts=1.
2019-09-05T17:58:03.000 controller-0 sm: debug time[7709.951] log<1602> INFO: sm[107278]: sm_service_disabling_state.c(259): Service (ceph-mon) is failed and has reached max action failures (4).
2019-09-05T17:58:03.000 controller-0 sm: debug time[7709.951] log<1603> INFO: sm[107278]: sm_service_fsm.c(1032): Service (ceph-mon) received event (disable-failed) was in the disabling state and is now in the disabling-failed state, condition=action-failure.
2019-09-05T17:58:03.000 controller-0 sm: debug time[7709.951] log<1604> INFO: sm[107278]: sm_service_group_audit.c(434): Service group (controller-services) member (ceph-mon) has mapped status (failed), overall_status=failed.
2019-09-05T17:58:03.000 controller-0 sm: debug time[7709.951] log<1605> INFO: sm[107278]: sm_service_group_audit.c(444): Service group (controller-services) member (ceph-mon) has mapped condition (action-failure), overall_condition=action-failure.
2019-09-05T17:58:04.000 controller-0 sm: debug time[7710.583] log<1612> INFO: sm[107278]: sm_service_api.c(300): Recovery of service (ceph-mon) requested.
2019-09-05T17:58:04.000 controller-0 sm: debug time[7710.847] log<1613> INFO: sm[107278]: sm_service_engine.c(347): Fail counts and status for service (ceph-mon) cleared in disabling state.
2019-09-05T17:58:04.000 controller-0 sm: debug time[7710.848] log<1614> INFO: sm[107278]: sm_service_enable.c(461): Started enable action (1032745) for service (ceph-mon).
2019-09-05T17:58:04.000 controller-0 sm: debug time[7710.848] log<1615> INFO: sm[107278]: sm_service_fsm.c(1032): Service (ceph-mon) received event (enable-throttle) was in the disabling-failed state and is now in the enabling-failed state.
2019-09-05T17:58:07.000 controller-0 sm: debug time[7714.365] log<1643> INFO: sm[107278]: sm_service_enable.c(477): Aborting action (enable) of service (ceph-mon).
2019-09-05T17:58:07.000 controller-0 sm: debug time[7714.365] log<1644> INFO: sm[107278]: sm_service_action.c(345): Aborting service (ceph-mon) with kill signal, pid=1032745.
2019-09-05T17:58:07.000 controller-0 sm: debug time[7714.366] log<1645> INFO: sm[107278]: sm_service_disable.c(380): Started disable action (1034053) for service (ceph-mon), flag (0).
2019-09-05T17:58:07.000 controller-0 sm: debug time[7714.366] log<1646> INFO: sm[107278]: sm_service_fsm.c(1032): Service (ceph-mon) received event (disable) was in the enabling-failed state and is now in the disabling-failed state.