Changing kube-apiserver oidc params causes apiserver's advertise-address to be on OAM IP

Bug #1900153 reported by Andy on 2020-10-16
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Low
Jerry Sun

Bug Description

Brief Description
-----------------
When kube-apiserver oidc service parameters get changed, the kube-apiserver is restarted with the --advertise-address updated to cluster's OAM IP address. The apiserver's advertise-address shouldn't be changed (should remain on cluster-host IP address).

Severity
--------
Major
(this behavior will cause issues during OAM IP change)

Steps to Reproduce
------------------
1. Before any oidc param change, make a copy of /etc/kubernetes/manifests/kube-apiserver.yaml, eg
   cp --parents /etc/kubernetes/manifests/kube-apiserver.yaml .

2. Check kube-apiserver's advertise-address, the --advertise-address=abcd:205::2, which is the cluster-host IP address.

[root@controller-0 patches(keystone_admin)]# ps -ef | grep apiserver
root 100567 100363 6 01:02 ? 00:47:07 kube-apiserver --advertise-address=abcd:205::2 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --default-not-ready-toleration-seconds=30 --default-unreachable-toleration-seconds=30 --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --encryption-provider-config=/etc/kubernetes/encryption-provider.yaml --etcd-servers=http://[abcd:205::1]:2379 --event-ttl=24h --feature-gates=SCTPSupport=true,TTLAfterFinished=true,HugePageStorageMediumSize=true --insecure-port=0 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-cluster-ip-range=abcd:207::/112 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key

3. Add oidc service parameters and apply them:

system service-parameter-add kubernetes kube_apiserver oidc_issuer_url=https://[2620:10a:a001:a103::165]:30556/dex oidc_username_claim=email oidc_client_id=stx-oidc-client-app

system service-parameter-apply

4. Check again the kube-apiserver's advertise-addres, it's now "--advertise-address=2620:10a:a001:a103::165", which is the cluster's OAM IP address.

[root@controller-0 patches(keystone_admin)]# ps -ef | grep apiserver
root 1121147 1121094 8 14:06 ? 00:01:31 kube-apiserver --advertise-address=2620:10a:a001:a103::165 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --default-not-ready-toleration-seconds=30 --default-unreachable-toleration-seconds=30 --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --encryption-provider-config=/etc/kubernetes/encryption-provider.yaml --etcd-servers=http://[abcd:205::1]:2379 --event-ttl=24h --feature-gates=SCTPSupport=true,TTLAfterFinished=true,HugePageStorageMediumSize=true --insecure-port=0 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --oidc-client-id=stx-oidc-client-app --oidc-issuer-url=https://[2620:10a:a001:a103::165]:30556/dex --oidc-username-claim=email --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-cluster-ip-range=abcd:207::/112 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key

5. Check the changes to /etc/kubernetes/manifests/kube-apiserver.yaml, the advertise-address.endpoint and host attributes are changed to OAM IP address.

[root@controller-0 patches(keystone_admin)]# diff ./etc/kubernetes/manifests/kube-apiserver.yaml /etc/kubernetes/manifests/kube-apiserver.yaml
5c5
< kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: '[abcd:205::2]:6443'
---
> kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: '[2620:10a:a001:a103::165]:6443'
16c16
< - --advertise-address=abcd:205::2
---
> - --advertise-address=2620:10a:a001:a103::165
31a32,34
> - --oidc-client-id=stx-oidc-client-app
> - --oidc-issuer-url=https://[2620:10a:a001:a103::165]:30556/dex
> - --oidc-username-claim=email
49c52
< host: abcd:205::2
---
> host: 2620:10a:a001:a103::165

Expected Behavior
------------------
kube-apiserver's advertise-address should remain on cluster-host IP.

Actual Behavior
----------------
kube-apiserver's advertise-address is updated to the cluster's OAM IP.

Reproducibility
---------------
Reproducible

System Configuration
--------------------
This is observed on a IPv6 DX lab, but I think it will happen on IPv4 as well.

Branch/Pull Time/Commit
-----------------------
stx master

Last Pass
---------
Unknown

Timestamp/Logs
--------------
See steps to reproduce.

Test Activity
-------------
Developer Testing

Workaround
----------
Manually update /etc/kubernetes/manifests/kube-apiserver.yaml to restore advertise-address.endpoint and host IP.

Andy (andy.wrs) wrote :

The root cause of the issue seem to be in kube-apiserver-change-params.erb.

The command "kubeadm --kubeconfig=/etc/kubernetes/admin.conf config view > <%= @configmap_temp_file %> " dump the kube-apiserver config in a file, then the command "kubeadm init phase control-plane apiserver --config <%= @configmap_temp_file %>" take that config file, update /etc/kubernetes/manifests/kube-apiserver.yaml, and restart apiserver. But the command doesn't specify advertise-address, and the dumped config file doesn't have advertise-address settings so kubeadm uses default network interface.

Ghada Khalil (gkhalil) wrote :

Low priority - This is an issue when changing oidc and oam at the same time, so the likelihood of occurrence is low. Will mark for stx.5.0 for now time-permitting.

Changed in starlingx:
importance: Undecided → Medium
status: New → Triaged
tags: added: stx.apps
tags: added: stx.5.0
Changed in starlingx:
importance: Medium → Low
assignee: nobody → Jerry Sun (jerry-sun-u)
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers