kube-apiserver endpoints not configured correctly
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
StarlingX |
Fix Released
|
High
|
Matt Peters |
Bug Description
Brief Description
-----------------
The kube-apiserver endpoints are being configured incorrectly:
- controller-0 endpoint set to floating cluster IP
- controller-1 endpoint set to OAM unit IP
The endpoints can change when controllers are locked/unlocked and re-installed.
The correct configuration would be:
- controller-0 endpoint set to controller-0 cluster IP
- controller-1 endpoint set to controller-1 cluster IP
Severity
--------
Major: kube-apiserver endpoints should not be on OAM network and should be fixed to the controller they are running on.
Steps to Reproduce
------------------
Install a lab.
Expected Behavior
------------------
See above.
Actual Behavior
----------------
It looks like this was always broken and got worse when we moved to using “kubadm join” on the second controller instead of “kubadm init”.
In short:
- controller-0 (ansible bootstrap does “kubeadm init” with kubeadm.yaml config file):
- sets the InitConfiguration localAPIEndpoin
- sets the ClusterConfigur
- controller-1 (runs “kubeadm join” using cluster configuration):
- the InitConfiguration localAPIEndpoin
Using the WC-4 as an example this results in:
# kubectl -n kube-system get configmap kubeadm-config -o yaml
ClusterStatus: |
apiEndpoints:
controller-0:
bindPort: 6443
controller-1:
bindPort: 6443
apiVersion: kubeadm.
kind: ClusterStatus
# kubectl get ep kubernetes
NAME ENDPOINTS AGE
kubernetes [2620:10a:
When looking at the endpoints, both API servers are using the local OAM IPs, which doesn’t line up with the config map. I believe this is explained here:
https:/
// LocalAPIEndpoint represents the endpoint of the API server instance that's deployed on this control plane node
// In HA setups, this differs from ClusterConfigur
// is the global endpoint for the cluster, which then loadbalances the requests to each individual API server. This
// configuration object lets you customize what IP/DNS name and port the local API server advertises it's accessible
// on. By default, kubeadm tries to auto-detect the IP of the default interface and use that, but in case that process
// fails you may set the desired value here.
LocalAPIEnd
Not sure of the fix yet, but I think:
- At bootstrap we need to set InitConfiguration localAPIEndpoin
- When doing the join on controller-1 (or on controller-0 reinstall), we need to pass the --apiserver-
There may be other changes required - we need to check that the static manifests in /etc/kubernetes
Reproducibility
---------------
Reproducible
System Configuration
-------
All configurations are affected
Branch/Pull Time/Commit
-------
stx.4.0 load built from master on 2020-05-05
Last Pass
---------
Unknown
Timestamp/Logs
--------------
See above
Test Activity
-------------
Developer Testing
Workaround
----------
None
tags: | added: stx.4.0 stx.containers |
Changed in starlingx: | |
importance: | Undecided → High |
status: | New → Triaged |
assignee: | nobody → Frank Miller (sensfan22) |
Changed in starlingx: | |
assignee: | Paul-Ionut Vaduva (pvaduva) → Matt Peters (mpeters-wrs) |
Changed in starlingx: | |
status: | Triaged → In Progress |
Related fix proposed to branch: master /review. opendev. org/726231
Review: https:/