Subcloud is deleted but shows alarm at system controller for subcloud offline

[Takamasa Takenaka]

Subcloud is deleted but shows alarm at system controller for subcloud offline

Brief Description
-----------------
Subcloud is deleted properly but shows alarm at system controller
for subcloud offline.

Severity
--------
<Minor: System/Feature is usable with minor issue>

Steps to Reproduce
------------------
1. Put subcloud offline (The alarm list shows subcloud is offline))
2. Delete subcloud
3. Offline alarm stays in the alarm list

Expected Behavior
------------------
Subcloud offline alarm is removed from the alarm list.

Actual Behavior
----------------
Subcloud offline alarm stays in the alarm list.

Reproducibility
---------------
100% reproducible if fm manager is restarted.

System Configuration
--------------------
DC

Branch/Pull Time/Commit
-----------------------
master
N/A

Timestamp/Logs
--------------
N/A

Test Activity
-------------
Feature Testing

Workaround
----------
No work around

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/fault/+/898144
Committed: https://opendev.org/starlingx/fault/commit/8bd6e5b92d50e15235f2777ea715aaea75c2886e
Submitter: "Zuul (22348)"
Branch: master

commit 8bd6e5b92d50e15235f2777ea715aaea75c2886e
Author: Takamasa Takenaka <email address hidden>
Date: Thu Oct 12 19:21:44 2023 -0300

    Validate fm manager socket fd before send a message

    When fm manager is restarted, there is no mechanism to detect it
    from fm api client side. As a result, when subcloud delete clear
    alarm request is sent after fm manager is restarted, fm api client
    will show broke pipe and clear alarm request is not received and
    this alarm stays.

    This fix is to check socket fd state before send/receive from
    fm api client. If broken pipe is detected, it will try to
    reconnect to fm manager.

    Closes-bug: 2039684

    Test Plan:
    PASS: Restart fm manager and confirm that detect broken pipe
          and reconnect messages in /var/log. For example,
    -----
    sm: err fmSocket.cpp(270): A broken pipe error occurred
    sm: warning fmAPI.cpp(116): Invalid file descriptor. Atte
    mpting to reconnect...
    sm: info fmAPI.cpp(149): Connected to FM Manager.
    -----
    PASS: Delete offline subcloud and confirm the alarm is
          removed.

    Change-Id: Ibc0f4d96b5c0a385d8fedbc1acd23898f1cbea46
    Signed-off-by: Takamasa Takenaka <email address hidden>