I took a look at the lab, and found that both subcloud 1 & 6 out-of-sync were caused by the same error.
The dcmanager failed to notify the dcorch about the subcloud state change due to an exception from the dcorch. The exception was originated from the dbsyncclient due to ‘Unauthorized request’.
There is no retry for RPC notification and no more subcloud state change to trigger a new notification. As a result, the subcloud 1 & 6 were offline in the dcorch and sync states were not updated.
After un-manage/manage subcloud 1 &6, the system recovered.
2020-01-09 17:29:37.444 115738 INFO dcmanager.manager.subcloud_audit_manager [-] Setting new availability status: online on subcloud: subcloud1
2020-01-09 17:29:37.970 115738 ERROR dcmanager.manager.subcloud_audit_manager [-] Remote error: Unauthorized Unauthorized request.
2020-01-09 17:29:37.973 115738 WARNING dcmanager.manager.subcloud_audit_manager [-] Problem informing dcorch of subcloud state change, subcloud: subcloud1: RemoteError: Remote error: Unauthorized Unauthorized request.
[u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/usr/lib/python2.7/site-packages/dcorch/engine/service.py", line 48, in wrapped\n return func(self, ctx, *args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/dcorch/engine/service.py", line 204, in update_subcloud_states\n self.gsm.initial_sync(ctxt, subcloud_name)\n', u' File "/usr/lib/python2.7/site-packages/dcorch/engine/generic_sync_manager.py", line 116, in initial_sync\n subcloud_engine.initial_sync()\n', u' File "/usr/lib/python2.7/site-packages/dcorch/engine/subcloud.py", line 138, in initial_sync\n thread.initial_sync()\n', u' File "/usr/lib/python2.7/site-packages/dcorch/engine/sync_services/identity.py", line 239, in initial_sync\n consts.RESOURCE_TYPE_IDENTITY_USERS)\n', u' File "/usr/lib/python2.7/site-packages/dcorch/engine/sync_services/identity.py", line 1525, in get_subcloud_resources\n self.sc_dbs_client)\n', u' File "/usr/lib/python2.7/site-packages/dcorch/engine/sync_services/identity.py", line 1095, in _get_resource_audit_handler\n return self._get_users_resource(client.identity_manager)\n', u' File "/usr/lib/python2.7/site-packages/dcorch/engine/sync_services/identity.py", line 1136, in _get_users_resource\n users = client.list_users()\n', u' File "/usr/lib/python2.7/site-packages/dcdbsync/dbsyncclient/v1/identity/identity_manager.py", line 192, in list_users\n return self.users_list(url)\n', u' File "/usr/lib/python2.7/site-packages/dcdbsync/dbsyncclient/v1/identity/identity_manager.py", line 108, in users_list\n raise exceptions.Unauthorized(\'Unauthorized request.\')\n', u'Unauthorized: Unauthorized request.\n'].: RemoteError: Remote error: Unauthorized Unauthorized request.
I took a look at the lab, and found that both subcloud 1 & 6 out-of-sync were caused by the same error.
The dcmanager failed to notify the dcorch about the subcloud state change due to an exception from the dcorch. The exception was originated from the dbsyncclient due to ‘Unauthorized request’.
There is no retry for RPC notification and no more subcloud state change to trigger a new notification. As a result, the subcloud 1 & 6 were offline in the dcorch and sync states were not updated.
After un-manage/manage subcloud 1 &6, the system recovered.
2020-01-09 17:29:37.444 115738 INFO dcmanager. manager. subcloud_ audit_manager [-] Setting new availability status: online on subcloud: subcloud1 manager. subcloud_ audit_manager [-] Remote error: Unauthorized Unauthorized request. manager. subcloud_ audit_manager [-] Problem informing dcorch of subcloud state change, subcloud: subcloud1: RemoteError: Remote error: Unauthorized Unauthorized request. python2. 7/site- packages/ oslo_messaging/ rpc/server. py", line 160, in _process_incoming\n res = self.dispatcher .dispatch( message) \n', u' File "/usr/lib/ python2. 7/site- packages/ oslo_messaging/ rpc/dispatcher. py", line 213, in dispatch\n return self._do_ dispatch( endpoint, method, ctxt, args)\n', u' File "/usr/lib/ python2. 7/site- packages/ oslo_messaging/ rpc/dispatcher. py", line 183, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/usr/lib/ python2. 7/site- packages/ dcorch/ engine/ service. py", line 48, in wrapped\n return func(self, ctx, *args, **kwargs)\n', u' File "/usr/lib/ python2. 7/site- packages/ dcorch/ engine/ service. py", line 204, in update_ subcloud_ states\ n self.gsm. initial_ sync(ctxt, subcloud_name)\n', u' File "/usr/lib/ python2. 7/site- packages/ dcorch/ engine/ generic_ sync_manager. py", line 116, in initial_sync\n subcloud_ engine. initial_ sync()\ n', u' File "/usr/lib/ python2. 7/site- packages/ dcorch/ engine/ subcloud. py", line 138, in initial_sync\n thread. initial_ sync()\ n', u' File "/usr/lib/ python2. 7/site- packages/ dcorch/ engine/ sync_services/ identity. py", line 239, in initial_sync\n consts. RESOURCE_ TYPE_IDENTITY_ USERS)\ n', u' File "/usr/lib/ python2. 7/site- packages/ dcorch/ engine/ sync_services/ identity. py", line 1525, in get_subcloud_ resources\ n self.sc_ dbs_client) \n', u' File "/usr/lib/ python2. 7/site- packages/ dcorch/ engine/ sync_services/ identity. py", line 1095, in _get_resource_ audit_handler\ n return self._get_ users_resource( client. identity_ manager) \n', u' File "/usr/lib/ python2. 7/site- packages/ dcorch/ engine/ sync_services/ identity. py", line 1136, in _get_users_ resource\ n users = client. list_users( )\n', u' File "/usr/lib/ python2. 7/site- packages/ dcdbsync/ dbsyncclient/ v1/identity/ identity_ manager. py", line 192, in list_users\n return self.users_ list(url) \n', u' File "/usr/lib/ python2. 7/site- packages/ dcdbsync/ dbsyncclient/ v1/identity/ identity_ manager. py", line 108, in users_list\n raise exceptions. Unauthorized( \'Unauthorized request.\')\n', u'Unauthorized: Unauthorized request.\n'].: RemoteError: Remote error: Unauthorized Unauthorized request.
2020-01-09 17:29:37.970 115738 ERROR dcmanager.
2020-01-09 17:29:37.973 115738 WARNING dcmanager.
[u'Traceback (most recent call last):\n', u' File "/usr/lib/