Comment 0 for bug 1951506

Brief Description
-----------------
After a subcloud was unmanaged and deleted, a dc-cert out-of-sync alarm (280.002) continued to be reported. The expected behavior is not see any alarms related to deleted subclouds as this creates a confusion for an operator

Severity
--------
Minor

Steps to Reproduce
------------------
- dcmanager subcloud unmanage
- dcmanager subcloud delete
- Check alarms

Expected Behavior
------------------
Alarms related to the deleted subcloud are removed

Actual Behavior
----------------
A dc-cert out-of-sync alarm is still present

Reproducibility
---------------
Intermittent

System Configuration
--------------------
Distributed Cloud

Branch/Pull Time/Commit
-----------------------
stx.5.0, but the same issue also exists in the main branch

Last Pass
---------
Unknown

Timestamp/Logs
--------------
cert-mon.log.* shows the following errors related to the subcloud even after a deletion:
Line 35098: 2021-10-02T00:03:02.000 controller-0 cert-mon: info 187857 INFO sysinv.cert_mon.watcher [-] action ADDED (subcloud1-adminep-ca-certificate)
Line 35101: 2021-10-02T00:03:02.000 controller-0 cert-mon: info 187857 INFO sysinv.cert_mon.watcher [-] subcloud subcloud1 action ADDED (subcloud1-adminep-ca-certificate)
Line 35101: 2021-10-02T00:03:02.000 controller-0 cert-mon: info 187857 INFO sysinv.cert_mon.watcher [-] subcloud subcloud1 action ADDED (subcloud1-adminep-ca-certificate)
Line 35104: 2021-10-02T00:03:03.000 controller-0 cert-mon: err 187857 ERROR sysinv.cert_mon.watcher [-] 13 Reattempt failed Cannot find sysinv endpoint for subcloud1. action ADDED (subcloud1-adminep-ca-certificate)
Line 35104: 2021-10-02T00:03:03.000 controller-0 cert-mon: err 187857 ERROR sysinv.cert_mon.watcher [-] 13 Reattempt failed Cannot find sysinv endpoint for subcloud1. action ADDED (subcloud1-adminep-ca-certificate)
Line 35106: created at 2021-09-29 22:44:57 last operation Update last update at 2021-09-29 22:47:55: Exception: Cannot find sysinv endpoint for subcloud1
Line 35107: 2021-10-02T00:03:03.000 controller-0 cert-mon: err 187857 ERROR sysinv.cert_mon.watcher [-] Cannot find sysinv endpoint for subcloud1: Exception: Cannot find sysinv endpoint for subcloud1
Line 35107: 2021-10-02T00:03:03.000 controller-0 cert-mon: err 187857 ERROR sysinv.cert_mon.watcher [-] Cannot find sysinv endpoint for subcloud1: Exception: Cannot find sysinv endpoint for subcloud1
Line 35119: 2021-10-02 00:03:03.292 187857 ERROR sysinv.cert_mon.watcher Exception: Cannot find sysinv endpoint for subcloud1
Line 35122: 2021-10-02T00:03:03.000 controller-0 cert-mon: info 187857 INFO sysinv.cert_mon.watcher [-] Attempt to update intermediate CA cert for subcloud1 has failed

Test Activity
-------------
Testing

Workaround
----------
Manually delete the alarm