kube-apiserver security parameters are overwritten on upgrades

[Kaustubh Dhokte]

The script here https://review.opendev.org/c/starlingx/integ/+/845654/9/kubernetes/kubernetes-1.22.5/centos/files/update-k8s-feature-gates.sh runs during platform upgrade, upgrade-activate phase which makes changes to the kube-apiserver feature-gates in the cluster configmap and updates its manifests. This template https://opendev.org/starlingx/stx-puppet/blame/branch/master/puppet-manifests/src/modules/platform/templates/kube-apiserver-change-params.erb updates some lines in the kube-apiserver manifests without updating the cluster configmap. The above script possibly overwrites those changes.
To be more specific, lines https://opendev.org/starlingx/stx-puppet/blame/branch/master/puppet-manifests/src/modules/platform/templates/kube-apiserver-change-params.erb#L50 and https://opendev.org/starlingx/stx-puppet/blame/branch/master/puppet-manifests/src/modules/platform/templates/kube-apiserver-change-params.erb#L51
could be overwritten due to the script above.

kube-apiserver-change-params.erb is executed as a part of service parameter apply and simplex upgrade.

Severity

Major

Steps to Reproduce

N/A

Expected Behavior

The script should not overwrite the specifies lines in the template.

Actual Behavior

Reproducibility

Reproducible

System Configuration

Load info (eg: 2022-03-10_20-00-07)

Last Pass

Use this section to also indicate if this is a new test scenario.

Timestamp/Logs

Alarms

If there are any alarms please list them here

Test Activity

Workaround

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/integ/+/853531

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/integ/+/853531
Committed: https://opendev.org/starlingx/integ/commit/73632416b3fc5ddaa8e2b4babb93ba00fd6c58ca
Submitter: "Zuul (22348)"
Branch: master

commit 73632416b3fc5ddaa8e2b4babb93ba00fd6c58ca
Author: Kaustubh Dhokte <email address hidden>
Date: Wed Aug 17 15:04:36 2022 -0400

    Preserve kube-apiserver manifest params during upgrade-activate

    This change https://opendev.org/starlingx/integ/commit/a6a5349d025487672fe818aae36a2020a9f9f08c
    (k8s-1.22.5: remove feature-gates)
    adds a script that is run during upgrade activate. The script modifies
    kubeadm cluster config and eventually updates kube-apiserver manifest
    to remove deprecated features-gates in k8s 1.22.

    As 'kubeadm init phase' is rerun in the script, it updates the
    kube-apiserver manifest to be in sync with the kubeadm cluster config.
    In that process, it nullifies the effect of these two commits,
    https://opendev.org/starlingx/stx-puppet/commit/04a1c1b0809f66488bd54e3f31d323430e7d9913
    (Rework advertise address in apiserver-change-param)
    and https://opendev.org/starlingx/stx-puppet/commit/52ace69c837acc7e3aff8a2d584968297afd70fe
    (Amend kube-apiserver 1.23 configuration to use PSP)

    This change adds a function to the script that preserves the effect
    of above listed commits.

    Test Plan:
    On CentOS AIO-SX
    PASS: Upgrade Successful. Check if advertise address in
          kube-apiserver manifest before and after running
          upgrade-activate is same.
          Ensure that the seccomp profile configuration is
          removed after upgrade-activate.
          Kube-apiserver is running and cluster is accessible after
          the upgrade.
    PASS: No Shellcheck errors

    Closes-Bug: 1986854

    Signed-off-by: Kaustubh Dhokte <email address hidden>
    Change-Id: Ib97e14bc5b4ed208e65e16888e1380a3bd9fdb8f

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/config/+/853923
Committed: https://opendev.org/starlingx/config/commit/3c8a992c5f8f685b0d5ad9fde8a679979ed7dba3
Submitter: "Zuul (22348)"
Branch: master

commit 3c8a992c5f8f685b0d5ad9fde8a679979ed7dba3
Author: Kaustubh Dhokte <email address hidden>
Date: Mon Aug 22 02:30:15 2022 -0400

    Add upgrade path for feature-gate removal

    The commit https://opendev.org/starlingx/integ/commit/a6a5349d025487672fe818aae36a2020a9f9f08c
    (k8s-1.22.5: remove feature-gates)
    added a script that removes deprecated feature gates which is run during
    upgrade-activate phase of previous upgrade cycle .

    The commit https://opendev.org/starlingx/integ/commit/73632416b3fc5ddaa8e2b4babb93ba00fd6c58ca
    (Preserve kube-apiserver manifest params during upgrade-activate)
    modified the script to preserve the kube-apiserver manifest parameters
    and it is supposed to run in next patch release upgrade.

    This change adds a new 'from_version' for the manifest to run during
    next patch release.

    The previous 'from_version' is still supported as in the future, we will
    need to support CentOS to Debian upgrade.

    Test Plan:
    On CentOS AIO-SX
    PASS: Upgrade Successful. Check if advertise address in
          kube-apiserver manifest before and after running
          upgrade-activate is same.
          Ensure that the seccomp profile configuration is
          removed after upgrade-activate.
          Kube-apiserver is running and cluster is accessible after
          the upgrade.

    Closes-Bug: 1986854

    Signed-off-by: Kaustubh Dhokte <email address hidden>
    Change-Id: I0e40df6e341f2da4f0e7ed4b4803197cd07470d5