Distributed Cloud - subcloud status out of sync during audit

[Brent Rowsell]

Brief Description
-----------------
It appears sub-clouds are marked out of sync every 10 mins and return to in sync state after 1-2 mins. In this case, there is no mismatch between the system controller and sub-clouds. This was seen on a DC system with 10 sub-clouds, all exhibited this behavior.

The sub-clouds should only ever be marked out of sync if there is a true out of sync condition. Audits should not mark out of sync and then in sync once if no mismatch exists.

Severity
--------
Minor

Steps to Reproduce
------------------
Setup DC system with a number of subclouds
Monitor the status of the subclouds

Expected Behavior
------------------
subclouds remain in sync

Actual Behavior
----------------
subclouds go out-of-sync

Reproducibility
---------------
seen multiple times

System Configuration
--------------------
Distributed Cloud
Lab Name: wolfpass-22-34

Branch/Pull Time/Commit
-----------------------
2019-11-06_10-52-51

Last Pass
---------
unknown

Timestamp/Logs
--------------
see Description

Test Activity
-------------
DC engineering

[Ghada Khalil (gkhalil)]

This sounds similar to https://bugs.launchpad.net/starlingx/+bug/1847661 which was already fixed

[Ghada Khalil (gkhalil)]

Low / not gating any stx release - no serious system impact

[Andy (andy.wrs)]

The load being used doesn't have the fix from: https://review.opendev.org/#/c/695309/

Without this fix, audit finds ssl_ca cert is out-of-sync so arrange a sync job for it. But the actual sync request failed silently and turn the sync status back to in-sync.

[Ghada Khalil (gkhalil)]

Marking as duplicate of https://bugs.launchpad.net/starlingx/+bug/1851252
Fix was merged on 2019-11-25 - review: https://review.opendev.org/#/c/695309/

[Ghada Khalil (gkhalil)]

Also changed the priority and release tag given that it was determined that this is a real issue with the certificate sync.