Logs show that the controller-1 was shutdown, due to ceph-mon fatal failure (exceed max transition failures). Then controller-0 went active as expected.
2019-08-14T09:01:13.000 controller-1 sm: debug time[1513.951] log<1946> INFO: sm[97172]: sm_service_audit.c(364): Service (dbmon) already running audit-disabled action.
2019-08-14T09:01:13.000 controller-1 sm: debug time[1514.051] log<1947> INFO: sm[97172]: sm_service_engine.c(157): Service (ceph-mon) has had a fatal failure and is unrecoverable.
2019-08-14T09:01:14.000 controller-1 sm: debug time[1514.555] log<1948> INFO: sm[97172]: sm_service_engine.c(157): Service (ceph-mon) has had a fatal failure and is unrecoverable.
2019-08-14T09:01:14.000 controller-1 sm: debug time[1515.056] log<1949> INFO: sm[97172]: sm_service_engine.c(157): Service (ceph-mon) has had a fatal failure and is unrecoverable.
2019-08-14T09:01:15.000 controller-1 sm: debug time[1515.558] log<1950> INFO: sm[97172]: sm_service_engine.c(157): Service (ceph-mon) has had a fatal failure and is unrecoverable.
2019-08-14T09:01:15.000 controller-1 sm: debug time[1515.917] log<1951> INFO: sm[97172]: sm_service_group_api.c(212): Service group (controller-services) recovery from fatal condition escalated to a reboot.
2019-08-14T09:01:15.000 controller-1 sm: debug time[1515.917] log<1952> INFO: sm[97172]: sm_node_api.cpp(964): Reboot of controller-1 requested, reason=service group (controller-services) recovery from fatal condition escalated to a reboot..
2019-08-14T09:01:15.000 controller-1 sm: debug time[1515.918] log<1953> INFO: sm[97172]: sm_troubleshoot.c(150): Troubleshoot process (401470) created.
ceph-mon errors:
2019-08-14T09:00:01.762 controller-1 OCF_drbd(drbd-cephmon:warning 1)[380462]: WARNING: drbd-cephmon still Primary, demoting.
2019-08-14T09:00:01.768 controller-1 OCF_drbd(drbd-cephmon:warning 1)[380462]: WARNING: drbd-cephmon is still mounted via /dev/drbd9
2019-08-14T09:00:01.771 controller-1 OCF_drbd(drbd-cephmon:warning 1)[380462]: WARNING: Waiting to drop drbd-cephmon
2019-08-14T09:00:02.777 controller-1 OCF_drbd(drbd-cephmon:warning 1)[380462]: WARNING: drbd-cephmon still Primary, demoting.
2019-08-14T09:00:02.783 controller-1 OCF_drbd(drbd-cephmon:warning 1)[380462]: WARNING: drbd-cephmon is still mounted via /dev/drbd9
2019-08-14T09:00:02.786 controller-1 OCF_drbd(drbd-cephmon:warning 1)[380462]: WARNING: Waiting to drop drbd-cephmon
2019-08-14T09:00:03.792 controller-1 OCF_drbd(drbd-cephmon:warning 1)[380462]: WARNING: drbd-cephmon still Primary, demoting.
2019-08-14T09:00:03.798 controller-1 OCF_drbd(drbd-cephmon:warning 1)[380462]: WARNING: drbd-cephmon is still mounted via /dev/drbd9
2019-08-14T09:00:03.800 controller-1 OCF_drbd(drbd-cephmon:warning 1)[380462]: WARNING: Waiting to drop drbd-cephmon
2019-08-14T09:00:04.806 controller-1 OCF_drbd(drbd-cephmon:warning 1)[380462]: WARNING: drbd-cephmon still Primary, demoting.
2019-08-14T09:00:04.811 controller-1 OCF_drbd(drbd-cephmon:warning 1)[380462]: WARNING: drbd-cephmon is still mounted via /dev/drbd9
2019-08-14T09:00:04.814 controller-1 OCF_drbd(drbd-cephmon:warning 1)[380462]: WARNING: Waiting to drop drbd-cephmon
2019-08-14T09:00:05.819 controller-1 OCF_drbd(drbd-cephmon:warning 1)[380462]: WARNING: drbd-cephmon still Primary, demoting.
2019-08-14T09:00:05.824 controller-1 OCF_drbd(drbd-cephmon:warning 1)[380462]: WARNING: drbd-cephmon is still mounted via /dev/drbd9
2019-08-14T09:00:05.827 controller-1 OCF_drbd(drbd-cephmon:warning 1)[380462]: WARNING: Waiting to drop drbd-cephmon
2019-08-14T09:00:06.833 controller-1 OCF_drbd(drbd-cephmon:warning 1)[380462]: WARNING: drbd-cephmon still Primary, demoting.
2019-08-14T09:00:06.839 controller-1 OCF_drbd(drbd-cephmon:warning 1)[380462]: WARNING: drbd-cephmon is still mounted via /dev/drbd9
2019-08-14T09:00:06.842 controller-1 OCF_drbd(drbd-cephmon:warning 1)[380462]: WARNING: Waiting to drop drbd-cephmon
Logs show that the controller-1 was shutdown, due to ceph-mon fatal failure (exceed max transition failures). Then controller-0 went active as expected.
| 2019-08- 14T08:54: 00.609 | 501 | service-scn | ceph-mon | enabling | disabling | enable failed 14T08:56: 01.831 | 502 | service-scn | ceph-mon | disabling | disabling-failed | disable failed 14T08:56: 01.831 | 503 | service-group-scn | controller-services | go-active | go-active-failed | ceph-mon(disabling, failed) 14T08:56: 02.256 | 504 | service-scn | ceph-mon | disabling-failed | enabling-failed | enabled-active state requested 14T08:56: 05.362 | 505 | service-group-scn | vim-services | active | disabling | 14T08:56: 05.363 | 506 | service-group-scn | cloud-services | active | disabling | 14T08:56: 05.364 | 507 | service-group-scn | controller-services | go-active-failed | disabling-failed | ceph-mon(enabling, failed)
| 2019-08-
| 2019-08-
| 2019-08-
| 2019-08-
| 2019-08-
| 2019-08-
2019-08- 14T09:01: 13.000 controller-1 sm: debug time[1513.951] log<1946> INFO: sm[97172]: sm_service_ audit.c( 364): Service (dbmon) already running audit-disabled action. 14T09:01: 13.000 controller-1 sm: debug time[1514.051] log<1947> INFO: sm[97172]: sm_service_ engine. c(157): Service (ceph-mon) has had a fatal failure and is unrecoverable. 14T09:01: 14.000 controller-1 sm: debug time[1514.555] log<1948> INFO: sm[97172]: sm_service_ engine. c(157): Service (ceph-mon) has had a fatal failure and is unrecoverable. 14T09:01: 14.000 controller-1 sm: debug time[1515.056] log<1949> INFO: sm[97172]: sm_service_ engine. c(157): Service (ceph-mon) has had a fatal failure and is unrecoverable. 14T09:01: 15.000 controller-1 sm: debug time[1515.558] log<1950> INFO: sm[97172]: sm_service_ engine. c(157): Service (ceph-mon) has had a fatal failure and is unrecoverable. 14T09:01: 15.000 controller-1 sm: debug time[1515.917] log<1951> INFO: sm[97172]: sm_service_ group_api. c(212): Service group (controller- services) recovery from fatal condition escalated to a reboot. 14T09:01: 15.000 controller-1 sm: debug time[1515.917] log<1952> INFO: sm[97172]: sm_node_ api.cpp( 964): Reboot of controller-1 requested, reason=service group (controller- services) recovery from fatal condition escalated to a reboot.. 14T09:01: 15.000 controller-1 sm: debug time[1515.918] log<1953> INFO: sm[97172]: sm_troubleshoot .c(150) : Troubleshoot process (401470) created.
2019-08-
2019-08-
2019-08-
2019-08-
2019-08-
2019-08-
2019-08-
ceph-mon errors: 14T09:00: 01.762 controller-1 OCF_drbd( drbd-cephmon: warning 1)[380462]: WARNING: drbd-cephmon still Primary, demoting. 14T09:00: 01.768 controller-1 OCF_drbd( drbd-cephmon: warning 1)[380462]: WARNING: drbd-cephmon is still mounted via /dev/drbd9 14T09:00: 01.771 controller-1 OCF_drbd( drbd-cephmon: warning 1)[380462]: WARNING: Waiting to drop drbd-cephmon 14T09:00: 02.777 controller-1 OCF_drbd( drbd-cephmon: warning 1)[380462]: WARNING: drbd-cephmon still Primary, demoting. 14T09:00: 02.783 controller-1 OCF_drbd( drbd-cephmon: warning 1)[380462]: WARNING: drbd-cephmon is still mounted via /dev/drbd9 14T09:00: 02.786 controller-1 OCF_drbd( drbd-cephmon: warning 1)[380462]: WARNING: Waiting to drop drbd-cephmon 14T09:00: 03.792 controller-1 OCF_drbd( drbd-cephmon: warning 1)[380462]: WARNING: drbd-cephmon still Primary, demoting. 14T09:00: 03.798 controller-1 OCF_drbd( drbd-cephmon: warning 1)[380462]: WARNING: drbd-cephmon is still mounted via /dev/drbd9 14T09:00: 03.800 controller-1 OCF_drbd( drbd-cephmon: warning 1)[380462]: WARNING: Waiting to drop drbd-cephmon 14T09:00: 04.806 controller-1 OCF_drbd( drbd-cephmon: warning 1)[380462]: WARNING: drbd-cephmon still Primary, demoting. 14T09:00: 04.811 controller-1 OCF_drbd( drbd-cephmon: warning 1)[380462]: WARNING: drbd-cephmon is still mounted via /dev/drbd9 14T09:00: 04.814 controller-1 OCF_drbd( drbd-cephmon: warning 1)[380462]: WARNING: Waiting to drop drbd-cephmon 14T09:00: 05.819 controller-1 OCF_drbd( drbd-cephmon: warning 1)[380462]: WARNING: drbd-cephmon still Primary, demoting. 14T09:00: 05.824 controller-1 OCF_drbd( drbd-cephmon: warning 1)[380462]: WARNING: drbd-cephmon is still mounted via /dev/drbd9 14T09:00: 05.827 controller-1 OCF_drbd( drbd-cephmon: warning 1)[380462]: WARNING: Waiting to drop drbd-cephmon 14T09:00: 06.833 controller-1 OCF_drbd( drbd-cephmon: warning 1)[380462]: WARNING: drbd-cephmon still Primary, demoting. 14T09:00: 06.839 controller-1 OCF_drbd( drbd-cephmon: warning 1)[380462]: WARNING: drbd-cephmon is still mounted via /dev/drbd9 14T09:00: 06.842 controller-1 OCF_drbd( drbd-cephmon: warning 1)[380462]: WARNING: Waiting to drop drbd-cephmon
2019-08-
2019-08-
2019-08-
2019-08-
2019-08-
2019-08-
2019-08-
2019-08-
2019-08-
2019-08-
2019-08-
2019-08-
2019-08-
2019-08-
2019-08-
2019-08-
2019-08-
2019-08-