Brief Description
-----------------
When kube-apiserver oidc service parameters get changed, the kube-apiserver is restarted with the --advertise-address updated to cluster's OAM IP address. The apiserver's advertise-address shouldn't be changed (should remain on cluster-host IP address).
Severity
--------
Major
(this behavior will cause issues during OAM IP change)
Steps to Reproduce
------------------
1. Before any oidc param change, make a copy of /etc/kubernetes/manifests/kube-apiserver.yaml, eg
cp --parents /etc/kubernetes/manifests/kube-apiserver.yaml .
2. Check kube-apiserver's advertise-address, the --advertise-address=abcd:205::2, which is the cluster-host IP address.
[root@controller-0 patches(keystone_admin)]# ps -ef | grep apiserver
root 100567 100363 6 01:02 ? 00:47:07 kube-apiserver --advertise-address=abcd:205::2 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --default-not-ready-toleration-seconds=30 --default-unreachable-toleration-seconds=30 --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --encryption-provider-config=/etc/kubernetes/encryption-provider.yaml --etcd-servers=http://[abcd:205::1]:2379 --event-ttl=24h --feature-gates=SCTPSupport=true,TTLAfterFinished=true,HugePageStorageMediumSize=true --insecure-port=0 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-cluster-ip-range=abcd:207::/112 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
3. Add oidc service parameters and apply them:
system service-parameter-add kubernetes kube_apiserver oidc_issuer_url=https://[2620:10a:a001:a103::165]:30556/dex oidc_username_claim=email oidc_client_id=stx-oidc-client-app
system service-parameter-apply
4. Check again the kube-apiserver's advertise-addres, it's now "--advertise-address=2620:10a:a001:a103::165", which is the cluster's OAM IP address.
[root@controller-0 patches(keystone_admin)]# ps -ef | grep apiserver
root 1121147 1121094 8 14:06 ? 00:01:31 kube-apiserver --advertise-address=2620:10a:a001:a103::165 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --default-not-ready-toleration-seconds=30 --default-unreachable-toleration-seconds=30 --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --encryption-provider-config=/etc/kubernetes/encryption-provider.yaml --etcd-servers=http://[abcd:205::1]:2379 --event-ttl=24h --feature-gates=SCTPSupport=true,TTLAfterFinished=true,HugePageStorageMediumSize=true --insecure-port=0 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --oidc-client-id=stx-oidc-client-app --oidc-issuer-url=https://[2620:10a:a001:a103::165]:30556/dex --oidc-username-claim=email --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-cluster-ip-range=abcd:207::/112 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
5. Check the changes to /etc/kubernetes/manifests/kube-apiserver.yaml, the advertise-address.endpoint and host attributes are changed to OAM IP address.
[root@controller-0 patches(keystone_admin)]# diff ./etc/kubernetes/manifests/kube-apiserver.yaml /etc/kubernetes/manifests/kube-apiserver.yaml
5c5
< kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: '[abcd:205::2]:6443'
---
> kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: '[2620:10a:a001:a103::165]:6443'
16c16
< - --advertise-address=abcd:205::2
---
> - --advertise-address=2620:10a:a001:a103::165
31a32,34
> - --oidc-client-id=stx-oidc-client-app
> - --oidc-issuer-url=https://[2620:10a:a001:a103::165]:30556/dex
> - --oidc-username-claim=email
49c52
< host: abcd:205::2
---
> host: 2620:10a:a001:a103::165
Expected Behavior
------------------
kube-apiserver's advertise-address should remain on cluster-host IP.
Actual Behavior
----------------
kube-apiserver's advertise-address is updated to the cluster's OAM IP.
Reproducibility
---------------
Reproducible
System Configuration
--------------------
This is observed on a IPv6 DX lab, but I think it will happen on IPv4 as well.
Branch/Pull Time/Commit
-----------------------
stx master
Last Pass
---------
Unknown
Timestamp/Logs
--------------
See steps to reproduce.
Test Activity
-------------
Developer Testing
Workaround
----------
Manually update /etc/kubernetes/manifests/kube-apiserver.yaml to restore advertise-address.endpoint and host IP.
The root cause of the issue seem to be in kube-apiserver- change- params. erb.
The command "kubeadm --kubeconfig= /etc/kubernetes /admin. conf config view > <%= @configmap_ temp_file %> " dump the kube-apiserver config in a file, then the command "kubeadm init phase control-plane apiserver --config <%= @configmap_ temp_file %>" take that config file, update /etc/kubernetes /manifests/ kube-apiserver. yaml, and restart apiserver. But the command doesn't specify advertise-address, and the dumped config file doesn't have advertise-address settings so kubeadm uses default network interface.