performance problems with k8s 1.21.3

[Chris Friesen]

K8s 1.21.3 is giving worse overhead than older versions of K8s (such as 1.18.1 as was used in StarlingX 5). In particular, the steady-state overhead of kubelet and kube-apiserver is higher when there are no pods on the system as well as when there are numerous pods on a node.

K8s 1.21.8 has been tested and gives much better behaviour, especially when there are many pods on a given node. As such we want to move to 1.21.8 instead of 1.21.3.

The main side effect of this is that we also need to bump to a slightly newer version of golang but it's only a patch version bump so nothing should be impacted.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/compile/+/824802

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/integ/+/824803

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/config/+/824804

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/824805

[Ghada Khalil (gkhalil)]

screening: stx.7.0 / medium - performance issue related to k8s and requiring a minor upversion. Shouldn't cause a major problem for the basic StarlingX use-cases

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/compile/+/824802
Committed: https://opendev.org/starlingx/compile/commit/1c369d087b9579baa06abc048d6181bf9da5cc60
Submitter: "Zuul (22348)"
Branch: master

commit 1c369d087b9579baa06abc048d6181bf9da5cc60
Author: Chris Friesen <email address hidden>
Date: Fri Jan 7 20:09:51 2022 -0500

    Replace golang 1.16.6 with 1.16.12

    In order to add kubernetes 1.21.8 we need to add support for
    golang 1.16.12. We're going to be removing kubernetes 1.21.3 so
    we no longer need golang 1.16.6.

    The ppc64le-vdso-fix.patch patch was merged upstream and so is
    no longer applicable.

    No direct testing is possible, but it is tested indirectly by
    building and testing Kubernetes 1.21.8.

    Partial-Bug: 1957994
    Signed-off-by: Chris Friesen <email address hidden>
    Change-Id: I25d18ee9c7e6a2a2aaa72c896b4687768804fa30

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/integ/+/824803
Committed: https://opendev.org/starlingx/integ/commit/df3b902e6bfbafe8114889323dcadfe1debf572e
Submitter: "Zuul (22348)"
Branch: master

commit df3b902e6bfbafe8114889323dcadfe1debf572e
Author: Chris Friesen <email address hidden>
Date: Fri Jan 7 20:49:31 2022 -0500

    Replace K8s 1.21.3 with 1.21.8

    In testing K8s 1.21.8 used less CPU than 1.21.3, so we are moving to
    the newer version.

    This has been booted in vbox and a basic pod has been started.
    A full regression will be performed.

    Depends-On: https://review.opendev.org/c/starlingx/compile/+/824802

    Partial-Bug: 1957994
    Signed-off-by: Chris Friesen <email address hidden>
    Change-Id: I64e4a64c90ef7591aeee52742dfcba9fdd8e5063

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/ansible-playbooks/+/824805
Committed: https://opendev.org/starlingx/ansible-playbooks/commit/393a2a3c8789760e3b3af655cb9b7d48040f10ff
Submitter: "Zuul (22348)"
Branch: master

commit 393a2a3c8789760e3b3af655cb9b7d48040f10ff
Author: Chris Friesen <email address hidden>
Date: Fri Jan 7 21:00:07 2022 -0500

    Replace K8s 1.21.3 with 1.21.8

    In testing K8s 1.21.8 used less CPU than 1.21.3, so we are moving to
    the newer version.

    This has been booted in vbox and a basic pod has been started.
    A full regression will be performed.

    Partial-Bug: 1957994
    Depends-On: https://review.opendev.org/c/starlingx/integ/+/824803
    Change-Id: If27099aac6852b8b50f1f6d8d37ddf27d6a108ac
    Signed-off-by: Chris Friesen <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/config/+/824804
Committed: https://opendev.org/starlingx/config/commit/f37403a9436cc43a7691b39c6ac98cf8c0711f4e
Submitter: "Zuul (22348)"
Branch: master

commit f37403a9436cc43a7691b39c6ac98cf8c0711f4e
Author: Chris Friesen <email address hidden>
Date: Fri Jan 7 21:16:09 2022 -0500

    Replace K8s 1.21.3 with 1.21.8

    In testing K8s 1.21.8 used less CPU than 1.21.3, so we are moving to
    the newer version.

    This has been booted in vbox and a basic pod has been started.
    A full regression will be performed.
    Still need to install with 1.20 to confirm that we can upgrade to
    1.21.8.

    Closes-Bug: 1957994
    Depends-On: https://review.opendev.org/c/starlingx/integ/+/824803
    Depends-On: https://review.opendev.org/c/starlingx/ansible-playbooks/+/824805
    Signed-off-by: Chris Friesen <email address hidden>
    Change-Id: I7c6a75ff9bbdddf1bfab3d2ea71a25224293d989