dcmanager receives 401 when polls for Kubernetes versions after root CA update

[Andy]

Brief Description
-----------------
After performing k8s root CA update, dcmanager audit failed when polling kubernetes version from subcloud through sysinv-api, receiving 401 error.

Severity
--------
Minor

Steps to Reproduce
------------------
- Update subcloud kubernetes root CA certificate.
- Wait dcmanager audit the subcloud

Expected Behavior
-----------------
dcmanager audit succeeds on the subcloud.

Actual Behavior
---------------
dcmanager audit fails on the subcloud with 401 error.

Reproducibility
---------------
100%

System Configuration
--------------------
DC system with subclouds.

Branch/Pull Time/Commit
-----------------------
stx latest.

Last Pass
--------
N/A

Timestamp/Logs
--------------
DC audit will encounter errors when it polls for kubernetes version:

2021-09-06 19:43:51.493 111784 WARNING cgtsclient.common.http [-] Request returned failure status.
2021-09-06 19:43:51.493 111784 ERROR dcmanager.audit.subcloud_audit_manager [-] Error in periodic subcloud audit loop: HTTPInternalServerError: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict(
{'Date': 'Mon, 06 Sep 2021 19:43:51 GMT', 'Content-Length': '129', 'Content-Type': 'application/json'}

)
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
2021-09-06 19:43:51.493 111784 ERROR dcmanager.audit.subcloud_audit_manager Traceback (most recent call last):
2021-09-06 19:43:51.493 111784 ERROR dcmanager.audit.subcloud_audit_manager File "/usr/lib/python2.7/site-packages/dcmanager/audit/subcloud_audit_manager.py", line 217, in periodic_subcloud_audit
2021-09-06 19:43:51.493 111784 ERROR dcmanager.audit.subcloud_audit_manager self._periodic_subcloud_audit_loop()
2021-09-06 19:43:51.493 111784 ERROR dcmanager.audit.subcloud_audit_manager File "/usr/lib/python2.7/site-packages/dcmanager/audit/subcloud_audit_manager.py", line 414, in _periodic_subcloud_audit_loop
2021-09-06 19:43:51.493 111784 ERROR dcmanager.audit.subcloud_audit_manager audit_kube_rootca_update)
2021-09-06 19:43:51.493 111784 ERROR dcmanager.audit.subcloud_audit_manager File "/usr/lib/python2.7/site-packages/dcmanager/audit/subcloud_audit_manager.py", line 303, in _get_audit_data
2021-09-06 19:43:51.493 111784 ERROR dcmanager.audit.subcloud_audit_manager kubernetes_audit_data = self.kubernetes_audit.get_regionone_audit_data()
2021-09-06 19:43:51.493 111784 ERROR dcmanager.audit.subcloud_audit_manager File "/usr/lib/python2.7/site-packages/dcmanager/audit/kubernetes_audit.py", line 93, in get_regionone_audit_data
2021-09-06 19:43:51.493 111784 ERROR dcmanager.audit.subcloud_audit_manager results_list = sysinv_client.get_kube_versions()
2021-09-06 19:43:51.493 111784 ERROR dcmanager.audit.subcloud_audit_manager File "/usr/lib/python2.7/site-packages/dccommon/drivers/openstack/sysinv_v1.py", line 743, in get_kube_versions
2021-09-06 19:43:51.493 111784 ERROR dcmanager.audit.subcloud_audit_manager return self.sysinv_client.kube_version.list()
2021-09-06 19:43:51.493 111784 ERROR dcmanager.audit.subcloud_audit_manager File "/usr/lib64/python2.7/site-packages/cgtsclient/v1/kube_version.py", line 25, in list
2021-09-06 19:43:51.493 111784 ERROR dcmanager.audit.subcloud_audit_manager return self._list(self._path(), 'kube_versions')
2021-09-06 19:43:51.493 111784 ERROR dcmanager.audit.subcloud_audit_manager File "/usr/lib64/python2.7/site-packages/cgtsclient/common/base.py", line 71, in _list
2021-09-06 19:43:51.493 111784 ERROR dcmanager.audit.subcloud_audit_manager _, body = self.api.json_request('GET', url)
2021-09-06 19:43:51.493 111784 ERROR dcmanager.audit.subcloud_audit_manager File "/usr/lib64/python2.7/site-packages/cgtsclient/common/http.py", line 269, in json_request
2021-09-06 19:43:51.493 111784 ERROR dcmanager.audit.subcloud_audit_manager method, **kwargs)
2021-09-06 19:43:51.493 111784 ERROR dcmanager.audit.subcloud_audit_manager File "/usr/lib64/python2.7/site-packages/cgtsclient/common/http.py", line 245, in _cs_request
2021-09-06 19:43:51.493 111784 ERROR dcmanager.audit.subcloud_audit_manager error_json.get('debuginfo'), *args)
2021-09-06 19:43:51.493 111784 ERROR dcmanager.audit.subcloud_audit_manager HTTPInternalServerError: (401)
2021-09-06 19:43:51.493 111784 ERROR dcmanager.audit.subcloud_audit_manager Reason: Unauthorized
2021-09-06 19:43:51.493 111784 ERROR dcmanager.audit.subcloud_audit_manager HTTP response headers: HTTPHeaderDict(
{'Date': 'Mon, 06 Sep 2021 19:43:51 GMT', 'Content-Length': '129', 'Content-Type': 'application/json'}

)
2021-09-06 19:43:51.493 111784 ERROR dcmanager.audit.subcloud_audit_manager HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}

Test Activity
------------
Developer Testing

Workaround
----------
Restart sysinv api (sm-restart service sysinv-inv)

[Ghada Khalil (gkhalil)]

screening: minor issue related to stx.6.0 feature: https://storyboard.openstack.org/#!/story/2008675

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/stx-puppet/+/808095

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/stx-puppet/+/808095
Committed: https://opendev.org/starlingx/stx-puppet/commit/d44a8080cf44b813b19ac71a63915c874f381dab
Submitter: "Zuul (22348)"
Branch: master

commit d44a8080cf44b813b19ac71a63915c874f381dab
Author: Andy Ning <email address hidden>
Date: Thu Sep 9 10:00:42 2021 -0400

    Restart sysinv API after k8s root CA update

    During k8s root CA certificate update, the certificates in admin.conf
    have been updated. Since both sysinv conductor and api cache k8s client
    that get credentials from admin.conf, both of they need to restart.
    (currently only conductor is restarted)

    Closes-Bug: 1943080
    Signed-off-by: Andy Ning <email address hidden>
    Change-Id: Ic92d6e95980b088108b175d0cafa905fda2ffb09