kube-apiserver endpoints not configured correctly

[Bart Wensley]

Brief Description
-----------------
The kube-apiserver endpoints are being configured incorrectly:
- controller-0 endpoint set to floating cluster IP
- controller-1 endpoint set to OAM unit IP

The endpoints can change when controllers are locked/unlocked and re-installed.

The correct configuration would be:
- controller-0 endpoint set to controller-0 cluster IP
- controller-1 endpoint set to controller-1 cluster IP

Severity
--------
Major: kube-apiserver endpoints should not be on OAM network and should be fixed to the controller they are running on.

Steps to Reproduce
------------------
Install a lab.

Expected Behavior
------------------
See above.

Actual Behavior
----------------
It looks like this was always broken and got worse when we moved to using “kubadm join” on the second controller instead of “kubadm init”.

In short:
- controller-0 (ansible bootstrap does “kubeadm init” with kubeadm.yaml config file):
  - sets the InitConfiguration localAPIEndpoint/advertiseAddress to the floating cluster IP (wrong)
  - sets the ClusterConfiguration controlPlaneEndpoint to the floating cluster IP (correct)
- controller-1 (runs “kubeadm join” using cluster configuration):
  - the InitConfiguration localAPIEndpoint/advertiseAddress is not set (not part of the ClusterConfiguration) so defaults to “the IP of the default interface”.

Using the WC-4 as an example this results in:

# kubectl -n kube-system get configmap kubeadm-config -o yaml
  ClusterStatus: |
    apiEndpoints:
      controller-0:
        advertiseAddress: aefd::1 <- floating cluster IP
        bindPort: 6443
      controller-1:
        advertiseAddress: 2620:10a:a001:a103::1199 <- controller-1 OAM IP
        bindPort: 6443
    apiVersion: kubeadm.k8s.io/v1beta2
    kind: ClusterStatus

# kubectl get ep kubernetes
NAME ENDPOINTS AGE
kubernetes [2620:10a:a001:a103::1182]:6443,[2620:10a:a001:a103::1199]:6443 30h <- controller-0 and controller-1 OAM IPs

When looking at the endpoints, both API servers are using the local OAM IPs, which doesn’t line up with the config map. I believe this is explained here:
https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2

    // LocalAPIEndpoint represents the endpoint of the API server instance that's deployed on this control plane node
    // In HA setups, this differs from ClusterConfiguration.ControlPlaneEndpoint in the sense that ControlPlaneEndpoint
    // is the global endpoint for the cluster, which then loadbalances the requests to each individual API server. This
    // configuration object lets you customize what IP/DNS name and port the local API server advertises it's accessible
    // on. By default, kubeadm tries to auto-detect the IP of the default interface and use that, but in case that process
    // fails you may set the desired value here.
    LocalAPIEndpoint APIEndpoint `json:"localAPIEndpoint,omitempty"`

Not sure of the fix yet, but I think:
- At bootstrap we need to set InitConfiguration localAPIEndpoint/advertiseAddress to the controller-0 cluster IP (not floating IP).
- When doing the join on controller-1 (or on controller-0 reinstall), we need to pass the --apiserver-advertise-address parameter with the unit specific cluster IP. From the docs: --apiserver-advertise-address string: If the node should host a new control plane instance, the IP address the API Server will advertise it’s listening on. If not set the default network interface will be used.

There may be other changes required - we need to check that the static manifests in /etc/kubernetes/manifests have the right --advertise-address set for the kube-apiserver and that these changes are preserved over reboots/re-installs.

Reproducibility
---------------
Reproducible

System Configuration
--------------------
All configurations are affected

Branch/Pull Time/Commit
-----------------------
stx.4.0 load built from master on 2020-05-05

Last Pass
---------
Unknown

Timestamp/Logs
--------------
See above

Test Activity
-------------
Developer Testing

Workaround
----------
None

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/726231

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/726231
Committed: https://git.openstack.org/cgit/starlingx/ansible-playbooks/commit/?id=26fd273cf5175ba4bdd31d6b6b777814f1a6c860
Submitter: Zuul
Branch: master

commit 26fd273cf5175ba4bdd31d6b6b777814f1a6c860
Author: Matt Peters <email address hidden>
Date: Thu May 7 14:29:02 2020 -0500

    Add kube-apiserver port to calico failsafe rules

    An invalid GlobalNetworkPolicy or NetworkPolicy may prevent
    calico-node from communicating with the kube-apiserver.
    Once the communication is broken, calico-node is no longer
    able to update the policies since it cannot communicate to
    read the updated policies. It can also prevent the pod
    from starting since the policies will prevent it from
    reading the configuration.

    To ensure that this scenario does not happen, the kube-apiserver
    port is being added to the failsafe rules to ensure communication
    is always possible, regardless of the network policy configuration.

    Change-Id: I1b065a74e7ad0ba9b1fdba4b63136b97efbe98ce
    Closes-Bug: 1877166
    Related-Bug: 1877383
    Signed-off-by: Matt Peters <email address hidden>

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: f/centos8
Review: https://review.opendev.org/729809

[Frank Miller (sensfan22)]

Assigning to Paul to prime the proper solution for this LP. Please consult with Bart as needed.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/729809
Committed: https://git.openstack.org/cgit/starlingx/ansible-playbooks/commit/?id=73027425d4501a6b7785e91024c9e8ddbc03115d
Submitter: Zuul
Branch: f/centos8

commit 55c9afd075194f7669fa2a87e546f61034679b04
Author: Dan Voiculeasa <email address hidden>
Date: Wed May 13 14:19:52 2020 +0300

    Restore: disconnect etcd from ceph

    At the moment etcd is restored only if ceph data is kept.
    Etcd should be restored regardless if ceph data is kept or wiped.

    Story: 2006770
    Task 39751
    Change-Id: I9dfb1be0a83c3fdc5f1b29cbb974c5e0e2236ad3
    Signed-off-by: Dan Voiculeasa <email address hidden>

commit 003ddff574c74adf11cf8e4758e93ba0eed45a6a
Author: Don Penney <email address hidden>
Date: Fri May 8 11:35:58 2020 -0400

    Add playbook for updating static images

    This commit introduces a new playbook, upgrade-static-images.yml, used
    for downloading updating images and pushing to the local registry.

    Change-Id: I8884440261a5a4e27b40398e5a75c9d03b09d4ba
    Story: 2006781
    Task: 39706
    Signed-off-by: Don Penney <email address hidden>

commit 26fd273cf5175ba4bdd31d6b6b777814f1a6c860
Author: Matt Peters <email address hidden>
Date: Thu May 7 14:29:02 2020 -0500

    Add kube-apiserver port to calico failsafe rules

    An invalid GlobalNetworkPolicy or NetworkPolicy may prevent
    calico-node from communicating with the kube-apiserver.
    Once the communication is broken, calico-node is no longer
    able to update the policies since it cannot communicate to
    read the updated policies. It can also prevent the pod
    from starting since the policies will prevent it from
    reading the configuration.

    To ensure that this scenario does not happen, the kube-apiserver
    port is being added to the failsafe rules to ensure communication
    is always possible, regardless of the network policy configuration.

    Change-Id: I1b065a74e7ad0ba9b1fdba4b63136b97efbe98ce
    Closes-Bug: 1877166
    Related-Bug: 1877383
    Signed-off-by: Matt Peters <email address hidden>

commit bd0f14a7dfb206ccaa3ce0f5e7d9034703b3403c
Author: Robert Church <email address hidden>
Date: Tue May 5 15:11:15 2020 -0400

    Provide an update strategy for Tiller deployment

    In the case of a simplex controller configuration the current patching
    strategy for the Tiller environment will fail as the tiller ports will
    be in use when the new deployment is attempted to be applied. The
    resulting tiller pod will be stuck in a Pending state.

    This will be observed if the node becomes ready after 'helm init'
    installs the initial deployment and before the deployment is patched for
    environment checks.

    The deployment strategy provided by 'helm init' is unspecified. This
    change will allow one additional pod (current + new) and one unavailable
    pod (current) during an update. The maxUnavailable setting allows the
    tiller pod to be deleted which will release its ports, thus allowing the
    patch deployment to spin up an new pod to a Running state.

    Change-Id: I83c43c52a77...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/734865

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/734879

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/734865
Committed: https://git.openstack.org/cgit/starlingx/ansible-playbooks/commit/?id=966bce1398dd23be49acc9c55e833987148db454
Submitter: Zuul
Branch: master

commit 966bce1398dd23be49acc9c55e833987148db454
Author: Matt Peters <email address hidden>
Date: Mon Jun 8 09:59:01 2020 -0500

    Fix kubernetes apiserver advertise address

    Set the kube-apiserver advertise address to the local
    controller cluster-host unit address to ensure kubeadm
    does not attempt to discover the default address which
    in most cases will be invalid.

    Set the kubeadm InitConfiguration advertiseAddress for
    the initial controller master node.

    Closes-Bug: 1877383
    Change-Id: I759234685966234bf987a9e06be77a5f793ee782
    Signed-off-by: Matt Peters <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/734879
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=601612676e5883ed220ae86fb40dd1a02c76139c
Submitter: Zuul
Branch: master

commit 601612676e5883ed220ae86fb40dd1a02c76139c
Author: Matt Peters <email address hidden>
Date: Mon Jun 8 09:55:55 2020 -0500

    Fix kubernetes apiserver advertise address

    Set the kube-apiserver advertise address to the local
    controller cluster-host unit address to ensure kubeadm
    does not attempt to discover the default address which
    in most cases will be invalid.

    The option --apiserver-advertise-addres is added to the
    join command for controller nodes joining as master nodes.

    Closes-Bug: 1877383
    Depends-On: https://review.opendev.org/734865
    Change-Id: I1575da6d28d08731a8aaf4200f920f5e8f510fa0
    Signed-off-by: Matt Peters <email address hidden>