Taking a closer look at the code, we do trigger an audit for all resources each time a subcloud transitions from unmanaged to managed. In this case, it looks like the problem was that the patch/load audits were run before the initial sync for the subcloud was done and the initial sync (which updates keystone data in the subcloud) caused the token for the subcloud to be invalidated.
Here is where the subcloud was managed:
2020-08-15T18:54:39.000 controller-0 -sh: info HISTORY: PID=3334493 UID=42425 dcmanager --os-username 'admin' --os-password 'Li69nux*' --os-tenant-name admin --os-auth-url http://[fd01:11::2]:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL subcloud manage subcloud1
The initial sync was done here:
2020-08-15 18:54:50.185 204600 INFO dcorch.engine.initial_sync_manager [-] Initial sync for subcloud subcloud1
2020-08-15 18:54:54.750 204600 INFO dcorch.engine.generic_sync_manager [-] updating state for subcloud subcloud1 - management_state: None availability_status: None initial_sync_state: completed
The patch audit started while the initial sync was being done and failed due to a 401 Unauthorized response from keystone in the subcloud:
2020-08-15 18:54:52.533 204590 INFO dcmanager.audit.patch_audit [-] Triggered patch audit for subcloud: subcloud1.
2020-08-15 18:54:52.653 204590 ERROR dccommon.drivers.openstack.patching_v1 [-] query failed with RC: 401
2020-08-15 18:54:52.653 204590 WARNING dcmanager.audit.patch_audit [-] Cannot retrieve patches for subcloud: subcloud1, skip patch audit: Exception: query failed with RC: 401
Looking at subcloud2, which was managed later, the same problem doesn't happen.
Here is where the subcloud was managed:
2020-08-15T19:34:24.000 controller-0 -sh: info HISTORY: PID=3818089 UID=42425 dcmanager --os-username 'admin' --os-password 'Li69nux*' --os-tenant-name admin --os-auth-url http://[fd01:11::2]:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL subcloud manage subcloud2
The initial sync was done here:
2020-08-15 19:34:26.165 204600 INFO dcorch.engine.initial_sync_manager [-] Initial sync for subcloud subcloud2
2020-08-15 19:34:30.652 204600 INFO dcorch.engine.generic_sync_manager [-] updating state for subcloud subcloud2 - management_state: None availability_status: None initial_sync_state: completed
The patch audit was done after the initial sync completed - the audit was successful:
2020-08-15 19:34:30.677 204590 INFO dcmanager.audit.patch_audit [-] Triggered patch audit for subcloud: subcloud2.
The solution here may be to delay the patch/load/firmware audits that are triggered after the manage, to give the initial sync time to complete.
Taking a closer look at the code, we do trigger an audit for all resources each time a subcloud transitions from unmanaged to managed. In this case, it looks like the problem was that the patch/load audits were run before the initial sync for the subcloud was done and the initial sync (which updates keystone data in the subcloud) caused the token for the subcloud to be invalidated.
Here is where the subcloud was managed: 15T18:54: 39.000 controller-0 -sh: info HISTORY: PID=3334493 UID=42425 dcmanager --os-username 'admin' --os-password 'Li69nux*' --os-tenant-name admin --os-auth-url http://[fd01:11: :2]:5000/ v3 --os-user- domain- name Default --os-project- domain- name Default --os-endpoint-type internalURL subcloud manage subcloud1
2020-08-
The initial sync was done here: engine. initial_ sync_manager [-] Initial sync for subcloud subcloud1 engine. generic_ sync_manager [-] updating state for subcloud subcloud1 - management_state: None availability_ status: None initial_sync_state: completed
2020-08-15 18:54:50.185 204600 INFO dcorch.
2020-08-15 18:54:54.750 204600 INFO dcorch.
The patch audit started while the initial sync was being done and failed due to a 401 Unauthorized response from keystone in the subcloud: audit.patch_ audit [-] Triggered patch audit for subcloud: subcloud1. drivers. openstack. patching_ v1 [-] query failed with RC: 401 audit.patch_ audit [-] Cannot retrieve patches for subcloud: subcloud1, skip patch audit: Exception: query failed with RC: 401
2020-08-15 18:54:52.533 204590 INFO dcmanager.
2020-08-15 18:54:52.653 204590 ERROR dccommon.
2020-08-15 18:54:52.653 204590 WARNING dcmanager.
Looking at subcloud2, which was managed later, the same problem doesn't happen.
Here is where the subcloud was managed: 15T19:34: 24.000 controller-0 -sh: info HISTORY: PID=3818089 UID=42425 dcmanager --os-username 'admin' --os-password 'Li69nux*' --os-tenant-name admin --os-auth-url http://[fd01:11: :2]:5000/ v3 --os-user- domain- name Default --os-project- domain- name Default --os-endpoint-type internalURL subcloud manage subcloud2
2020-08-
The initial sync was done here: engine. initial_ sync_manager [-] Initial sync for subcloud subcloud2 engine. generic_ sync_manager [-] updating state for subcloud subcloud2 - management_state: None availability_ status: None initial_sync_state: completed
2020-08-15 19:34:26.165 204600 INFO dcorch.
2020-08-15 19:34:30.652 204600 INFO dcorch.
The patch audit was done after the initial sync completed - the audit was successful: audit.patch_ audit [-] Triggered patch audit for subcloud: subcloud2.
2020-08-15 19:34:30.677 204590 INFO dcmanager.
The solution here may be to delay the patch/load/firmware audits that are triggered after the manage, to give the initial sync time to complete.