Brief Description
-----------------
Provide a brief description of the issue. Usually, it should not be more than 2 to 3 lines.
Subcloud offline alarms remain raised despite the subcloud availability being online. The is observed when a subcloud comes online while swacting.
Severity
--------
Provide the severity of the defect.
Major: Unable to create patch strategy because of management affecting alarms
Steps to Reproduce
------------------
1. Power off subclouds
2. initiate swact while subclouds are becoming online
Expected Behavior
------------------
Offline alarm should be cleared
Actual Behavior
----------------
Offline alarm is not cleared
Reproducibility
---------------
<Reproducible/Intermittent/Seen once>
Seen once. Reproducible with a induced FM failure while subclouds become online.
System Configuration
--------------------
DC
Branch/Pull Time/Commit
-----------------------
Master (Oct. 23th 2023)
Last Pass
---------
Unknown
Timestamp/Logs
--------------
Attach the logs for debugging (use attachments in Launchpad or for large collect files use:
audit.log.1.gz:2023-09-25 13:48:29.857 217770 ERROR dcmanager.audit.alarm_aggregation [-] Failed to update alarms for subcloud678 error: Error communicating with https://[2620:10a:a001:ac12::54c2]:18003/v1/alarms/summary Request to https://[2620:10a:a001:ac12::54c2]:18003/v1/alarms/summary timed out: fmclient.common.exceptions.InvalidEndpoint: Error communicating with https://[2620:10a:a001:ac12::54c2]:18003/v1/alarms/summary Request to https://[2620:10a:a001:ac12::54c2]:18003/v1/alarms/summary timed out
audit.log.1.gz:2023-09-25 13:48:29.919 217769 ERROR dcmanager.audit.subcloud_audit_worker_manager [-] Failed to get OS Client for subcloud: subcloud929: keystoneauth1.exceptions.http.ServiceUnavailable: Service Unavailable (HTTP 503)
state.log:2023-09-25 13:48:33.099 217767 WARNING root [req-2ce74f0c-a8ef-4509-8740-86162fd0ba74 - - - - -] fm_python_extension: Failed to connect to FM manager
state.log:fmAPI.cpp(140): Failed to connect to FM Manager.
state.log:2023-09-25 13:48:35.045 217766 WARNING root [req-2ce74f0c-a8ef-4509-8740-86162fd0ba74 - - - - -] fm_python_extension: Failed to connect to FM manager
state.log:fmAPI.cpp(140): Failed to connect to FM Manager.
audit.log.1.gz:2023-09-25 13:49:00.097 217769 ERROR dccommon.drivers.openstack.fm [-] get_alarm_summary exception=Error communicating with https://[2620:10a:a001:ac12::2ec2]:18003/v1/alarms/summary Request to https://[2620:10a:a001:ac12::2ec2]:18003/v1/alarms/summary timed out: fmclient.common.exceptions.InvalidEndpoint: Error communicating with https://[2620:10a:a001:ac12::2ec2]:18003/v1/alarms/summary Request to https://[2620:10a:a001:ac12::2ec2]:18003/v1/alarms/summary timed out
state.log:2023-09-25 13:49:29.230 217767 ERROR dcmanager.state.subcloud_state_manager [req-b1f5cc10-9512-4ecc-a289-b02bef115e14 - - - - -] Problem informing dcorch of subcloud state change,subcloud: subcloud990: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 3ebeff9de5914143a1dfb9861f008aab
state.log:2023-09-25 13:49:31.169 217766 ERROR dcmanager.state.subcloud_state_manager [req-2ce74f0c-a8ef-4509-8740-86162fd0ba74 - - - - -] Problem informing dcorch of subcloud state change,subcloud: subcloud990: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID a005530a921341c7a24343fc34c46408
Test Activity
-------------
Feature Testing
Workaround
----------
Delete alarm manually
Fix proposed to branch: master /review. opendev. org/c/starlingx /distcloud/ +/899175
Review: https:/