Setting System parameter kube_apiserver admission_plugins to include PodSecurityPolicy deletes the kube-api pod

[ayyappa]

Brief Description
-----------------
Setting System parameter kube_apiserver admission_plugins to include PodSecurityPolicy deletes the kube-api pod on active controller

Severity
--------
Major

Steps to Reproduce
------------------
1)Set the kubernetes kube_apiserver admission_plugins system parameter to include PodSecurityPolicy

       system service-parameter-add kubernetes kube_apiserver admission_plugins=PodSecurityPolicy
2)Apply the Kubernetes system parameters

       system service-parameter-apply kubernetes

3)Kube-api pod gets deleted on the system
[sysadmin@controller-0 ~(keystone_admin)]$ kubectl get pods -n kube-system | grep api

Expected Behavior
------------------
kube-apiserver pod should not gets deleted after adding psp

Actual Behavior
----------------
kube-apiserver pod gets deleted after adding psp to system admission_plugins

Reproducibility
---------------
100%

System Configuration
--------------------
tested on all the following systems
simplex system,wcp_112 ipv6
duplex system, ip_18_19,ipv4

Branch/Pull Time/Commit
-----------------------
2020-05-31 20:01:53 -0400

Last Pass
---------
This is a new test scenario

Timestamp/Logs
--------------
2020-06-01 11:47:13.956

Test Activity
-------------
Feature Testing

Workaround
----------
Delete the system service parameter and apply the kube service brings back the pod

[Jerry Sun (jerry-sun-u)]

I tried this on a system I had up, it seems the pod is down but the apiserver process is still running somehow

[sysadmin@controller-0 ~(keystone_admin)]$ kubectl get pods --all-namespaces | grep apiserver
kube-system kube-apiserver-controller-1 1/1 Running 0 134m
[sysadmin@controller-0 ~(keystone_admin)]$ ps -ef | grep apiserver
root 3720596 3720575 5 14:04 ? 00:07:13 kube-apiserver --advertise-address=10.10.10.4 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --default-not-ready-toleration-seconds=30 --default-unreachable-toleration-seconds=30 --enable-admission-plugins=PodSecurityPolicy,NodeRestriction --enable-bootstrap-token-auth=true --encryption-provider-config=/etc/kubernetes/encryption-provider.yaml --etcd-servers=http://192.168.206.1:2379 --event-ttl=24h --feature-gates=SCTPSupport=true,TTLAfterFinished=true,HugePageStorageMediumSize=true --insecure-port=0 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-cluster-ip-range=10.96.0.0/12 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
sysadmin 4179067 4177470 0 16:19 pts/0 00:00:00 grep --color=auto apiserver
[sysadmin@controller-0 ~(keystone_admin)]$

[Ghada Khalil (gkhalil)]

Lowering the priority given that the api-server was still functional even though the pod doesn't look like it recovered.

[Ghada Khalil (gkhalil)]

Moving to stx.5.0 as there is no functional impact.

[Ghada Khalil (gkhalil)]

lowering the priority further as we're close to the stx.5.0 release date and there is no functional impact. Still would be nice to investigate and address if possible.

[Carmen Rata (crata)]

The issue is a known problem that was posted in https://github.com/kubernetes/kubernetes/issues/85952.

The fix involves adding RoleBinding configuration that binds "privileged-psp-user" role to the kubelet user. With this fix we make sure to allow the apserver pod creation by granting permission to the kubelet user to create that pod.
The privileged PSP and the "privileged-psp-user" role configurations already exist.
The new configuration will be added to psp-policies.yaml.

Code review issued: https://review.opendev.org/c/starlingx/ansible-playbooks/+/785890.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/ansible-playbooks/+/785890
Committed: https://opendev.org/starlingx/ansible-playbooks/commit/05d3d8c21f5017b26466bfd1cdd1f1e7accf266f
Submitter: "Zuul (22348)"
Branch: master

commit 05d3d8c21f5017b26466bfd1cdd1f1e7accf266f
Author: Carmen Rata <email address hidden>
Date: Mon Apr 12 10:35:57 2021 -0400

    Fix kube-apiserver pod removal

    This commit adds rolebinding configuration to bind
    the "privileged-psp-user" role to the kubelet user.
    It fixes the issue where the kube-apiserver pod does
    not get recreated after enabling PodSecurityPolicy
    plugin. With this fix we make sure to allow the apiserver
    pod creation by granting permission to the kubelet user
    to create that pod.

    Closes-Bug: 1881605

    Signed-off-by: Carmen Rata <email address hidden>
    Change-Id: Ibdf6d4cacf2ce83dfa744455dac460027b2a6e47

[Carmen Rata (crata)]

Fix was merged in commit: review.opendev.org / starlingx / ansible-playbooks / 05d3d8c21f5017b26466bfd1cdd1f1e7accf266f

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/792195

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/794324

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Chuck Short <email address hidden>" on branch: f/centos8
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/792195

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/ansible-playbooks/+/794324
Committed: https://opendev.org/starlingx/ansible-playbooks/commit/163ec9989cc7360dba4c572b2c43effd10306048
Submitter: "Zuul (22348)"
Branch: f/centos8

commit 4e96b762f549aadb0291cc9bcf3352ae923e94eb
Author: Mihnea Saracin <email address hidden>
Date: Sat May 22 15:48:19 2021 +0000

    Revert "Restore host filesystems with collected sizes"

    This reverts commit 255488739efa4ac072424b19f2dbb7a3adb0254e.

    Reason for revert: Did a rework to fix https://bugs.launchpad.net/starlingx/+bug/1926591. The original problem was in puppet, and this fix in ansible was not good enough, it generated some other problems.

    Change-Id: Iea79701a874effecb7fe995ac468d50081d1a84f
    Depends-On: I55ae6954d24ba32e40c2e5e276ec17015d9bba44

commit c064aacc377c8bd5336ceab825d4bcbf5af0b5e8
Author: Angie Wang <email address hidden>
Date: Fri May 21 21:28:02 2021 -0400

    Ensure apiserver keys are present before extract from tarball

    This is to fix the upgrade playbook issue that happens during
    AIO-SX upgrade from stx4.0 to stx5.0 which introduced by
    https://review.opendev.org/c/starlingx/ansible-playbooks/+/792093.
    The apiserver keys are not available in stx4.0 side so we need
    to ensure the keys under /etc/kubernetes/pki are present in the
    backed-up tarball before extracting, otherwise playbook fails
    because the keys are not found in the archive.

    Change-Id: I8602f07d1b1041a7fd3fff21e6f9a422b9784ab5
    Closes-Bug: 928925
    Signed-off-by: Angie Wang <email address hidden>

commit 0261f22ff7c23d2a8608fe3b51725c9f29931281
Author: Don Penney <email address hidden>
Date: Thu May 20 23:09:07 2021 -0400

    Update SX to DX migration to wait for coredns config

    This commit updates the SX to DX migration playbook to wait after
    modifying the system mode to duplex until the runtime manifest that
    updates coredns config has completed. The playbook will wait for up to
    20 minutes to allow for the possibilty that sysinv has multiple
    runtime manifests queued up, each of which could take several minutes.

    Depends-On: https://review.opendev.org/c/starlingx/stx-puppet/+/792494
    Depends-On: https://review.opendev.org/c/starlingx/config/+/792496
    Change-Id: I3bf94d3493ae20eeb16b3fdcb27576ee18c0dc4d
    Closes-Bug: 1929148
    Signed-off-by: Don Penney <email address hidden>

commit 7c4f17bd0d92fc1122823211e1c9787829d206a9
Author: Daniel Safta <email address hidden>
Date: Wed May 19 09:08:16 2021 +0000

    Fixed missing apiserver-etcd-client certs

    When controller-1 is the active controller
    the backup archive does not contain
    /etc/etcd/apiserver-etcd-client.{crt, key}

    This change adds a new task which brings
    the certs from /etc/kubernetes/pki

    Closes-bug: 1928925
    Signed-off-by: Daniel Safta <email address hidden>
    Change-Id: I3c68377603e1af9a71d104e5b1108e9582497a09

commit e221ef8fbe51aa6ca229b584fb5632fe512ad5cb
Author: David Sullivan <email address hidden>
Date: Wed May 19 16:01:27 2021 -0500

    Support boo...