Changing kube-apiserver oidc params causes apiserver's advertise-address to be on OAM IP

[Andy]

Brief Description
-----------------
When kube-apiserver oidc service parameters get changed, the kube-apiserver is restarted with the --advertise-address updated to cluster's OAM IP address. The apiserver's advertise-address shouldn't be changed (should remain on cluster-host IP address).

Severity
--------
Major
(this behavior will cause issues during OAM IP change)

Steps to Reproduce
------------------
1. Before any oidc param change, make a copy of /etc/kubernetes/manifests/kube-apiserver.yaml, eg
   cp --parents /etc/kubernetes/manifests/kube-apiserver.yaml .

2. Check kube-apiserver's advertise-address, the --advertise-address=abcd:205::2, which is the cluster-host IP address.

[root@controller-0 patches(keystone_admin)]# ps -ef | grep apiserver
root 100567 100363 6 01:02 ? 00:47:07 kube-apiserver --advertise-address=abcd:205::2 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --default-not-ready-toleration-seconds=30 --default-unreachable-toleration-seconds=30 --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --encryption-provider-config=/etc/kubernetes/encryption-provider.yaml --etcd-servers=http://[abcd:205::1]:2379 --event-ttl=24h --feature-gates=SCTPSupport=true,TTLAfterFinished=true,HugePageStorageMediumSize=true --insecure-port=0 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-cluster-ip-range=abcd:207::/112 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key

3. Add oidc service parameters and apply them:

system service-parameter-add kubernetes kube_apiserver oidc_issuer_url=https://[2620:10a:a001:a103::165]:30556/dex oidc_username_claim=email oidc_client_id=stx-oidc-client-app

system service-parameter-apply

4. Check again the kube-apiserver's advertise-addres, it's now "--advertise-address=2620:10a:a001:a103::165", which is the cluster's OAM IP address.

[root@controller-0 patches(keystone_admin)]# ps -ef | grep apiserver
root 1121147 1121094 8 14:06 ? 00:01:31 kube-apiserver --advertise-address=2620:10a:a001:a103::165 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --default-not-ready-toleration-seconds=30 --default-unreachable-toleration-seconds=30 --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --encryption-provider-config=/etc/kubernetes/encryption-provider.yaml --etcd-servers=http://[abcd:205::1]:2379 --event-ttl=24h --feature-gates=SCTPSupport=true,TTLAfterFinished=true,HugePageStorageMediumSize=true --insecure-port=0 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --oidc-client-id=stx-oidc-client-app --oidc-issuer-url=https://[2620:10a:a001:a103::165]:30556/dex --oidc-username-claim=email --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-cluster-ip-range=abcd:207::/112 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key

5. Check the changes to /etc/kubernetes/manifests/kube-apiserver.yaml, the advertise-address.endpoint and host attributes are changed to OAM IP address.

[root@controller-0 patches(keystone_admin)]# diff ./etc/kubernetes/manifests/kube-apiserver.yaml /etc/kubernetes/manifests/kube-apiserver.yaml
5c5
< kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: '[abcd:205::2]:6443'
---
> kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: '[2620:10a:a001:a103::165]:6443'
16c16
< - --advertise-address=abcd:205::2
---
> - --advertise-address=2620:10a:a001:a103::165
31a32,34
> - --oidc-client-id=stx-oidc-client-app
> - --oidc-issuer-url=https://[2620:10a:a001:a103::165]:30556/dex
> - --oidc-username-claim=email
49c52
< host: abcd:205::2
---
> host: 2620:10a:a001:a103::165

Expected Behavior
------------------
kube-apiserver's advertise-address should remain on cluster-host IP.

Actual Behavior
----------------
kube-apiserver's advertise-address is updated to the cluster's OAM IP.

Reproducibility
---------------
Reproducible

System Configuration
--------------------
This is observed on a IPv6 DX lab, but I think it will happen on IPv4 as well.

Branch/Pull Time/Commit
-----------------------
stx master

Last Pass
---------
Unknown

Timestamp/Logs
--------------
See steps to reproduce.

Test Activity
-------------
Developer Testing

Workaround
----------
Manually update /etc/kubernetes/manifests/kube-apiserver.yaml to restore advertise-address.endpoint and host IP.

[Andy (andy.wrs)]

The root cause of the issue seem to be in kube-apiserver-change-params.erb.

The command "kubeadm --kubeconfig=/etc/kubernetes/admin.conf config view > <%= @configmap_temp_file %> " dump the kube-apiserver config in a file, then the command "kubeadm init phase control-plane apiserver --config <%= @configmap_temp_file %>" take that config file, update /etc/kubernetes/manifests/kube-apiserver.yaml, and restart apiserver. But the command doesn't specify advertise-address, and the dumped config file doesn't have advertise-address settings so kubeadm uses default network interface.

[Ghada Khalil (gkhalil)]

Low priority - This is an issue when changing oidc and oam at the same time, so the likelihood of occurrence is low. Will mark for stx.5.0 for now time-permitting.

[Ghada Khalil (gkhalil)]

Removing the stx.5.0 release tag as this is a low priority item.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/stx-puppet/+/821622

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/stx-puppet/+/821622
Committed: https://opendev.org/starlingx/stx-puppet/commit/4cb973737ea18656107eefd42ede7bdfbf549fce
Submitter: "Zuul (22348)"
Branch: master

commit 4cb973737ea18656107eefd42ede7bdfbf549fce
Author: Rafael Camargos <email address hidden>
Date: Mon Dec 13 16:50:58 2021 -0300

    Add advertise address to apiserver-change-params

    This work is fixing two issues:

    1. Executing the apiserver-change-params script will cause the
       kube-apiserver advertise-address parameter to be reset, resulting in
       a disruption in the communication with kubernetes. The cause of this
       bug is that kubeadm config view doesn't dump the advertise-address,
       which is a property of InitConfiguration, so using it only causes
       kubeadm init phase control-plane apiserver to set a default
       advertise-address.
    2. The command kubeadm config view became deprecated in k8s v1.19 and
       was removed in k8s v1.22 as reported on
       https://github.com/kubernetes/kubeadm/issues/2203.

    Test Plan: Verify advertise-address is not reset during apiserver change
    params

    PASS: Verify that modifying and applying kube-apiserver oidc parameters
    doesn't reset advertise-address in
    /etc/kubernetes/manifests/kube-apiserver.yaml
    PASS: Verify that upgrade command `system upgrade-activate` doesn't
    reset advertise-address in /etc/kubernetes/manifests/kube-apiserver.yaml
    PASS: Verify upgrade completes succesfully

    Regression: Verify other kube-apiserver resources are not affected

    PASS: Verify /etc/kubernetes/manifests/kube-apiserver.yaml changes in
    apiserver-change-params are consistent
    PASS: Verify ConfigMap ClusterConfiguration changes in
    apiserver-change-params are consistent
    PASS: Verify ConfigMap kubeadm-config changes in apiserver-change-params
    are consistent

    Notes: Above tests were executed in systems with both kubernetes v1.18.1
    and v1.21.3

    Closes-Bug: 1900153
    Signed-off-by: Rafael Camargos <email address hidden>
    Change-Id: Ic81a8e20990aa88aad240a7c6d9a79dee38aa370

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/stx-puppet/+/824232

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/stx-puppet/+/824232
Committed: https://opendev.org/starlingx/stx-puppet/commit/04a1c1b0809f66488bd54e3f31d323430e7d9913
Submitter: "Zuul (22348)"
Branch: master

commit 04a1c1b0809f66488bd54e3f31d323430e7d9913
Author: Rafael Camargos <email address hidden>
Date: Tue Jan 11 14:16:22 2022 -0300

    Rework advertise address in apiserver-change-param

    The current implementation assumes that /etc/kubernetes/kubeadm.yaml is
    always present on the system, which isn't true when the active host
    isn't controller-0.

    The new implementation will store the original advertise-address value
    and replace it on kube-apiserver config as soon as it gets reset during
    kubeadm init phase control-plane.

    The downside of this approach is that kube-apiserver is initially
    started with the default advertise address value (default network
    interface) to then get updated with the previous correct value,
    restarting kube-apiserver once more.

    Test Plan: Verify advertise-address is not affected during apiserver
    change params

    Note: advertise-address is found in
    `/etc/kubernetes/manifests/kube-apiserver.yaml`

    PASS: Verify that modifying and applying kube-apiserver oidc parameters
    doesn't affect advertise-address
    PASS: Verify upgrade command `system upgrade-activate` doesn't affect
    advertise-address
    PASS: Verify `/etc/kubernetes/manifests/kube-apiserver.yaml` changes after
    apiserver-change-params are consistent
    PASS: Verify ConfigMap kube-system kubeadm-config changes are consistent
    after apiserver-change-params are consistent after
    apiserver-change-params
    PASS: IPv6 - Verify that modifying kube-apiserver params doesn't affect
    advertise-address

    Regression:

    PASS: Verify system install
    PASS: Verify upgrade completes successfully
    PASS: Verify backup and restore completes successfully

    Notes: When possible, the above tests were executed in both AIO-SX and
    Standard systems, both controllers.

    Closes-Bug: 1900153
    Signed-off-by: Rafael Camargos <email address hidden>
    Change-Id: I95c5cc277fc343c383f9e4b3942e13f2009e1ab6