I checked /var/log/fm-event.log and see this problems:
2021-04-09T13:48:23.000 controller-1 fmManager: info { "event_log_id" : "750.006", "reason_text" : "A configuration change requires a reapply of the stx-openstack application.", "entity_instance_id" : "region=RegionOne.system=1606aba2-4182-485d-b46d-d50a67bdab33.k8s_application=stx-openstack", "severity" : "warning", "state" : "set", "timestamp" : "2021-04-09 13:48:23.531584" } 2021-04-09T13:48:25.000 controller-1 fmManager: info { "event_log_id" : "400.005", "reason_text" : "Communication failure detected with peer over port eno2 on host controller-1 within the last 30 seconds", "entity_instance_id" : "region=RegionOne.system=1606aba2-4182-485d-b46d-d50a67bdab33.host=controller-1.network=cluster-host", "severity" : "major", "state" : "clear", "timestamp" : "2021-04-09 13:48:25.799332" } 2021-04-09T13:48:25.000 controller-1 fmManager: info { "event_log_id" : "401.005", "reason_text" : "Communication established with peer over port eno2 on host controller-1", "entity_instance_id" : "host=controller-1.network=cluster-host", "severity" : "critical", "state" : "msg", "timestamp" : "2021-04-09 13:48:25.637745" } 2021-04-09T13:48:25.000 controller-1 fmManager: info { "event_log_id" : "401.005", "reason_text" : "Communication established with peer over port eno1 on host controller-1", "entity_instance_id" : "host=controller-1.network=oam", "severity" : "critical", "state" : "msg", "timestamp" : "2021-04-09 13:48:25.762306" } 2021-04-09T13:48:26.000 controller-1 fmManager: info { "event_log_id" : "400.005", "reason_text" : "Communication failure detected with peer over port eno1 on host controller-1 within the last 30 seconds", "entity_instance_id" : "region=RegionOne.system=1606aba2-4182-485d-b46d-d50a67bdab33.host=controller-1.network=oam", "severity" : "major", "state" : "clear", "timestamp" : "2021-04-09 13:48:26.001493" } 2021-04-09T13:48:26.000 controller-1 fmManager: info { "event_log_id" : "401.005", "reason_text" : "Communication established with peer over port eno2 on host controller-1", "entity_instance_id" : "host=controller-1.network=mgmt", "severity" : "critical", "state" : "msg", "timestamp" : "2021-04-09 13:48:25.845300" } 2021-04-09T13:48:26.000 controller-1 fmManager: info { "event_log_id" : "400.005", "reason_text" : "Communication failure detected with peer over port eno2 on host controller-1 within the last 30 seconds", "entity_instance_id" : "region=RegionOne.system=1606aba2-4182-485d-b46d-d50a67bdab33.host=controller-1.network=mgmt", "severity" : "major", "state" : "clear", "timestamp" : "2021-04-09 13:48:26.203367" } 2021-04-09T13:48:29.000 controller-1 fmManager: info { "event_log_id" : "401.001", "reason_text" : "Service group web-services state change from go-active to active on host controller-0", "entity_instance_id" : "service_domain=controller.service_group=web-services.host=controller-0", "severity" : "critical", "state" : "msg", "timestamp" : "2021-04-09 13:48:29.163167" } 2021-04-09T13:48:32.000 controller-1 fmManager: info { "event_log_id" : "401.002", "reason_text" : "Service group web-services redundancy restored", "entity_instance_id" : "service_domain=controller.service_group=web-services", "severity" : "critical", "state" : "msg", "timestamp" : "2021-04-09 13:48:31.888832" } 2021-04-09T13:48:32.000 controller-1 fmManager: info { "event_log_id" : "400.002", "reason_text" : "Service group web-services loss of redundancy; expected 2 active members but only 1 active member available", "entity_instance_id" : "region=RegionOne.system=1606aba2-4182-485d-b46d-d50a67bdab33.service_domain=controller.service_group=web-services", "severity" : "major", "state" : "clear", "timestamp" : "2021-04-09 13:48:32.008168" } 2021-04-09T13:49:11.000 controller-1 fmManager: info { "event_log_id" : "750.006", "reason_text" : "A configuration change requires a reapply of the stx-openstack application.", "entity_instance_id" : "region=RegionOne.system=1606aba2-4182-485d-b46d-d50a67bdab33.k8s_application=stx-openstack", "severity" : "warning", "state" : "clear", "timestamp" : "2021-04-09 13:49:11.024289" } 2021-04-09T13:49:11.000 controller-1 fmManager: info { "event_log_id" : "750.004", "reason_text" : "Application Apply In Progress", "entity_instance_id" : "region=RegionOne.system=1606aba2-4182-485d-b46d-d50a67bdab33.k8s_application=stx-openstack", "severity" : "warning", "state" : "set", "timestamp" : "2021-04-09 13:49:11.057328" } 2021-04-09T13:50:07.000 controller-1 fmManager: info { "event_log_id" : "750.004", "reason_text" : "Application Apply In Progress", "entity_instance_id" : "region=RegionOne.system=1606aba2-4182-485d-b46d-d50a67bdab33.k8s_application=stx-openstack", "severity" : "warning", "state" : "clear", "timestamp" : "2021-04-09 13:50:07.653780" } 2021-04-09T13:50:07.000 controller-1 fmManager: info { "event_log_id" : "750.002", "reason_text" : "Application Apply Failure", "entity_instance_id" : "region=RegionOne.system=1606aba2-4182-485d-b46d-d50a67bdab33.k8s_application=stx-openstack", "severity" : "major", "state" : "set", "timestamp" : "2021-04-09 13:50:07.598428" }
I checked /var/log/ fm-event. log and see this problems:
2021-04- 09T13:48: 23.000 controller-1 fmManager: info { "event_log_id" : "750.006", "reason_text" : "A configuration change requires a reapply of the stx-openstack application.", "entity_ instance_ id" : "region= RegionOne. system= 1606aba2- 4182-485d- b46d-d50a67bdab 33.k8s_ application= stx-openstack" , "severity" : "warning", "state" : "set", "timestamp" : "2021-04-09 13:48:23.531584" } 09T13:48: 25.000 controller-1 fmManager: info { "event_log_id" : "400.005", "reason_text" : "Communication failure detected with peer over port eno2 on host controller-1 within the last 30 seconds", "entity_ instance_ id" : "region= RegionOne. system= 1606aba2- 4182-485d- b46d-d50a67bdab 33.host= controller- 1.network= cluster- host", "severity" : "major", "state" : "clear", "timestamp" : "2021-04-09 13:48:25.799332" } 09T13:48: 25.000 controller-1 fmManager: info { "event_log_id" : "401.005", "reason_text" : "Communication established with peer over port eno2 on host controller-1", "entity_ instance_ id" : "host=controlle r-1.network= cluster- host", "severity" : "critical", "state" : "msg", "timestamp" : "2021-04-09 13:48:25.637745" } 09T13:48: 25.000 controller-1 fmManager: info { "event_log_id" : "401.005", "reason_text" : "Communication established with peer over port eno1 on host controller-1", "entity_ instance_ id" : "host=controlle r-1.network= oam", "severity" : "critical", "state" : "msg", "timestamp" : "2021-04-09 13:48:25.762306" } 09T13:48: 26.000 controller-1 fmManager: info { "event_log_id" : "400.005", "reason_text" : "Communication failure detected with peer over port eno1 on host controller-1 within the last 30 seconds", "entity_ instance_ id" : "region= RegionOne. system= 1606aba2- 4182-485d- b46d-d50a67bdab 33.host= controller- 1.network= oam", "severity" : "major", "state" : "clear", "timestamp" : "2021-04-09 13:48:26.001493" } 09T13:48: 26.000 controller-1 fmManager: info { "event_log_id" : "401.005", "reason_text" : "Communication established with peer over port eno2 on host controller-1", "entity_ instance_ id" : "host=controlle r-1.network= mgmt", "severity" : "critical", "state" : "msg", "timestamp" : "2021-04-09 13:48:25.845300" } 09T13:48: 26.000 controller-1 fmManager: info { "event_log_id" : "400.005", "reason_text" : "Communication failure detected with peer over port eno2 on host controller-1 within the last 30 seconds", "entity_ instance_ id" : "region= RegionOne. system= 1606aba2- 4182-485d- b46d-d50a67bdab 33.host= controller- 1.network= mgmt", "severity" : "major", "state" : "clear", "timestamp" : "2021-04-09 13:48:26.203367" } 09T13:48: 29.000 controller-1 fmManager: info { "event_log_id" : "401.001", "reason_text" : "Service group web-services state change from go-active to active on host controller-0", "entity_ instance_ id" : "service_ domain= controller. service_ group=web- services. host=controller -0", "severity" : "critical", "state" : "msg", "timestamp" : "2021-04-09 13:48:29.163167" } 09T13:48: 32.000 controller-1 fmManager: info { "event_log_id" : "401.002", "reason_text" : "Service group web-services redundancy restored", "entity_ instance_ id" : "service_ domain= controller. service_ group=web- services" , "severity" : "critical", "state" : "msg", "timestamp" : "2021-04-09 13:48:31.888832" } 09T13:48: 32.000 controller-1 fmManager: info { "event_log_id" : "400.002", "reason_text" : "Service group web-services loss of redundancy; expected 2 active members but only 1 active member available", "entity_ instance_ id" : "region= RegionOne. system= 1606aba2- 4182-485d- b46d-d50a67bdab 33.service_ domain= controller. service_ group=web- services" , "severity" : "major", "state" : "clear", "timestamp" : "2021-04-09 13:48:32.008168" } 09T13:49: 11.000 controller-1 fmManager: info { "event_log_id" : "750.006", "reason_text" : "A configuration change requires a reapply of the stx-openstack application.", "entity_ instance_ id" : "region= RegionOne. system= 1606aba2- 4182-485d- b46d-d50a67bdab 33.k8s_ application= stx-openstack" , "severity" : "warning", "state" : "clear", "timestamp" : "2021-04-09 13:49:11.024289" } 09T13:49: 11.000 controller-1 fmManager: info { "event_log_id" : "750.004", "reason_text" : "Application Apply In Progress", "entity_ instance_ id" : "region= RegionOne. system= 1606aba2- 4182-485d- b46d-d50a67bdab 33.k8s_ application= stx-openstack" , "severity" : "warning", "state" : "set", "timestamp" : "2021-04-09 13:49:11.057328" } 09T13:50: 07.000 controller-1 fmManager: info { "event_log_id" : "750.004", "reason_text" : "Application Apply In Progress", "entity_ instance_ id" : "region= RegionOne. system= 1606aba2- 4182-485d- b46d-d50a67bdab 33.k8s_ application= stx-openstack" , "severity" : "warning", "state" : "clear", "timestamp" : "2021-04-09 13:50:07.653780" } 09T13:50: 07.000 controller-1 fmManager: info { "event_log_id" : "750.002", "reason_text" : "Application Apply Failure", "entity_ instance_ id" : "region= RegionOne. system= 1606aba2- 4182-485d- b46d-d50a67bdab 33.k8s_ application= stx-openstack" , "severity" : "major", "state" : "set", "timestamp" : "2021-04-09 13:50:07.598428" }
2021-04-
2021-04-
2021-04-
2021-04-
2021-04-
2021-04-
2021-04-
2021-04-
2021-04-
2021-04-
2021-04-
2021-04-
2021-04-