I took a look at the lab, and found that both subcloud 1 & 6 out-of-sync were caused by the same error. 

The dcmanager failed to notify the dcorch about the subcloud state change due to an exception from the dcorch.  The exception was originated from the dbsyncclient due to ‘Unauthorized request’.  

There is no retry for RPC notification and no more subcloud state change to trigger a new notification. As a result, the subcloud 1 & 6 were offline in the dcorch and sync states were not updated.

After un-manage/manage subcloud 1 &6, the system recovered.

2020-01-09 17:29:37.444 115738 INFO dcmanager.manager.subcloud_audit_manager [-] Setting new availability status: online on subcloud: subcloud1
2020-01-09 17:29:37.970 115738 ERROR dcmanager.manager.subcloud_audit_manager [-] Remote error: Unauthorized Unauthorized request.
2020-01-09 17:29:37.973 115738 WARNING dcmanager.manager.subcloud_audit_manager [-] Problem informing dcorch of subcloud state change, subcloud: subcloud1: RemoteError: Remote error: Unauthorized Unauthorized request.
[u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming\n    res = self.dispatcher.dispatch(message)\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch\n    result = func(ctxt, **new_args)\n', u'  File "/usr/lib/python2.7/site-packages/dcorch/engine/service.py", line 48, in wrapped\n    return func(self, ctx, *args, **kwargs)\n', u'  File "/usr/lib/python2.7/site-packages/dcorch/engine/service.py", line 204, in update_subcloud_states\n    self.gsm.initial_sync(ctxt, subcloud_name)\n', u'  File "/usr/lib/python2.7/site-packages/dcorch/engine/generic_sync_manager.py", line 116, in initial_sync\n    subcloud_engine.initial_sync()\n', u'  File "/usr/lib/python2.7/site-packages/dcorch/engine/subcloud.py", line 138, in initial_sync\n    thread.initial_sync()\n', u'  File "/usr/lib/python2.7/site-packages/dcorch/engine/sync_services/identity.py", line 239, in initial_sync\n    consts.RESOURCE_TYPE_IDENTITY_USERS)\n', u'  File "/usr/lib/python2.7/site-packages/dcorch/engine/sync_services/identity.py", line 1525, in get_subcloud_resources\n    self.sc_dbs_client)\n', u'  File "/usr/lib/python2.7/site-packages/dcorch/engine/sync_services/identity.py", line 1095, in _get_resource_audit_handler\n    return self._get_users_resource(client.identity_manager)\n', u'  File "/usr/lib/python2.7/site-packages/dcorch/engine/sync_services/identity.py", line 1136, in _get_users_resource\n    users = client.list_users()\n', u'  File "/usr/lib/python2.7/site-packages/dcdbsync/dbsyncclient/v1/identity/identity_manager.py", line 192, in list_users\n    return self.users_list(url)\n', u'  File "/usr/lib/python2.7/site-packages/dcdbsync/dbsyncclient/v1/identity/identity_manager.py", line 108, in users_list\n    raise exceptions.Unauthorized(\'Unauthorized request.\')\n', u'Unauthorized: Unauthorized request.\n'].: RemoteError: Remote error: Unauthorized Unauthorized request.