Response to kubectl commands are throttled on controller-0 of subcloud

[Yuxing]

Brief Description
-----------------
1. It appears that ownership of .kube and its subdirectory is root:root
2. Due to this kubectl is unable to use local-cache and reaches out to api server to find appropriate APIs and groups.
2a. Changed the ownership of .kube and its subdirectories to sysadmin:sys_protected. kubectl response are normal without any throttling.
3. This causes overload on api-server which invokes throttling of the request to protect itself against the flood of the requests.

Severity
--------
Major

Steps to Reproduce
------------------
1. Log in to sysadmin in subcloud controller
2. Run command: "sysadmin@controller-0 ~(keystone_admin)]$ kubectl -n deployment get nodes,service,pods"

Expected Behavior
------------------
Console will return current nodes, service and pods info

Actual Behavior
----------------
Error message shown: "Throttling request took 1.142565451s, request: GET: ..."

Reproducibility
---------------
100% reproducible

System Configuration
--------------------
Distributed Cloud

Branch/Pull Time/Commit
-----------------------
Latest master

Last Pass
---------
No test availbe

Timestamp/Logs
--------------
2020-06-03

Test Activity
-------------
N/A

 Workaround
 ----------
Update the folder ownership

[Yuxing (yuxing)]

This is a new issue brings by updating kubenetes from v1.16 to v1.18. Similar issues found in:
https://github.com/kubernetes/kubernetes/issues/86462
https://github.com/kubernetes/kubernetes/issues/84169

Will try to modify the fold ownership to avoid that error

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/734951

[OpenStack Infra (hudson-openstack)]

Change abandoned by Yuxing Jiang (<email address hidden>) on branch: master
Review: https://review.opendev.org/734143

[OpenStack Infra (hudson-openstack)]

Change abandoned by Yuxing Jiang (<email address hidden>) on branch: master
Review: https://review.opendev.org/734951

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/734951
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=f721e8c5d7ceb2a3faba236ec69d5640ebe275bc
Submitter: Zuul
Branch: master

commit f721e8c5d7ceb2a3faba236ec69d5640ebe275bc
Author: Yuxing Jiang <email address hidden>
Date: Wed Jun 10 17:05:48 2020 -0400

    Monitor the owership of dir /home/sysadmin/.kube

    This commit checks the owner of the /home/sysadmin/.kube directory.
    The owner of this directory should be sysadmin: sys_protected for
    Kubenetes work properly. This function is added into the
    sysinv_conductor_monitor function in sysinv-conductor OCF script. The
    .kube directory is created by the kubectl command, and the owership
    may changed to root: root in subcloud controller for initialize
    helm before helm v3. Therefore this workaround should be deleted
    afterwards

    Test:
    Deploy a DC system, check the .kube directory ownership in subcloud
    controller after the controller is unlocked. The ownership should be
    sysadmin: sys_protected.

    Change-Id: I9be0cd558d4f1083ba2fa1f15b717f3de4eb64c7
    Closes-Bug: 1882100
    Signed-off-by: Yuxing Jiang <email address hidden>