performance problems with k8s 1.21.3

Bug #1957994 reported by Chris Friesen
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Medium
Chris Friesen

Bug Description

K8s 1.21.3 is giving worse overhead than older versions of K8s (such as 1.18.1 as was used in StarlingX 5). In particular, the steady-state overhead of kubelet and kube-apiserver is higher when there are no pods on the system as well as when there are numerous pods on a node.

K8s 1.21.8 has been tested and gives much better behaviour, especially when there are many pods on a given node. As such we want to move to 1.21.8 instead of 1.21.3.

The main side effect of this is that we also need to bump to a slightly newer version of golang but it's only a patch version bump so nothing should be impacted.

Chris Friesen (cbf123)
tags: added: platform
Changed in starlingx:
assignee: nobody → Chris Friesen (cbf123)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to compile (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/compile/+/824802

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to integ (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/integ/+/824803

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to config (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/config/+/824804

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ansible-playbooks (master)
Ghada Khalil (gkhalil)
tags: added: stx.containers
Revision history for this message
Ghada Khalil (gkhalil) wrote (last edit ):

screening: stx.7.0 / medium - performance issue related to k8s and requiring a minor upversion. Shouldn't cause a major problem for the basic StarlingX use-cases

Changed in starlingx:
importance: Undecided → Medium
tags: added: stx.7.0
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to compile (master)

Reviewed: https://review.opendev.org/c/starlingx/compile/+/824802
Committed: https://opendev.org/starlingx/compile/commit/1c369d087b9579baa06abc048d6181bf9da5cc60
Submitter: "Zuul (22348)"
Branch: master

commit 1c369d087b9579baa06abc048d6181bf9da5cc60
Author: Chris Friesen <email address hidden>
Date: Fri Jan 7 20:09:51 2022 -0500

    Replace golang 1.16.6 with 1.16.12

    In order to add kubernetes 1.21.8 we need to add support for
    golang 1.16.12. We're going to be removing kubernetes 1.21.3 so
    we no longer need golang 1.16.6.

    The ppc64le-vdso-fix.patch patch was merged upstream and so is
    no longer applicable.

    No direct testing is possible, but it is tested indirectly by
    building and testing Kubernetes 1.21.8.

    Partial-Bug: 1957994
    Signed-off-by: Chris Friesen <email address hidden>
    Change-Id: I25d18ee9c7e6a2a2aaa72c896b4687768804fa30

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to integ (master)

Reviewed: https://review.opendev.org/c/starlingx/integ/+/824803
Committed: https://opendev.org/starlingx/integ/commit/df3b902e6bfbafe8114889323dcadfe1debf572e
Submitter: "Zuul (22348)"
Branch: master

commit df3b902e6bfbafe8114889323dcadfe1debf572e
Author: Chris Friesen <email address hidden>
Date: Fri Jan 7 20:49:31 2022 -0500

    Replace K8s 1.21.3 with 1.21.8

    In testing K8s 1.21.8 used less CPU than 1.21.3, so we are moving to
    the newer version.

    This has been booted in vbox and a basic pod has been started.
    A full regression will be performed.

    Depends-On: https://review.opendev.org/c/starlingx/compile/+/824802

    Partial-Bug: 1957994
    Signed-off-by: Chris Friesen <email address hidden>
    Change-Id: I64e4a64c90ef7591aeee52742dfcba9fdd8e5063

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ansible-playbooks (master)

Reviewed: https://review.opendev.org/c/starlingx/ansible-playbooks/+/824805
Committed: https://opendev.org/starlingx/ansible-playbooks/commit/393a2a3c8789760e3b3af655cb9b7d48040f10ff
Submitter: "Zuul (22348)"
Branch: master

commit 393a2a3c8789760e3b3af655cb9b7d48040f10ff
Author: Chris Friesen <email address hidden>
Date: Fri Jan 7 21:00:07 2022 -0500

    Replace K8s 1.21.3 with 1.21.8

    In testing K8s 1.21.8 used less CPU than 1.21.3, so we are moving to
    the newer version.

    This has been booted in vbox and a basic pod has been started.
    A full regression will be performed.

    Partial-Bug: 1957994
    Depends-On: https://review.opendev.org/c/starlingx/integ/+/824803
    Change-Id: If27099aac6852b8b50f1f6d8d37ddf27d6a108ac
    Signed-off-by: Chris Friesen <email address hidden>

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to config (master)

Reviewed: https://review.opendev.org/c/starlingx/config/+/824804
Committed: https://opendev.org/starlingx/config/commit/f37403a9436cc43a7691b39c6ac98cf8c0711f4e
Submitter: "Zuul (22348)"
Branch: master

commit f37403a9436cc43a7691b39c6ac98cf8c0711f4e
Author: Chris Friesen <email address hidden>
Date: Fri Jan 7 21:16:09 2022 -0500

    Replace K8s 1.21.3 with 1.21.8

    In testing K8s 1.21.8 used less CPU than 1.21.3, so we are moving to
    the newer version.

    This has been booted in vbox and a basic pod has been started.
    A full regression will be performed.
    Still need to install with 1.20 to confirm that we can upgrade to
    1.21.8.

    Closes-Bug: 1957994
    Depends-On: https://review.opendev.org/c/starlingx/integ/+/824803
    Depends-On: https://review.opendev.org/c/starlingx/ansible-playbooks/+/824805
    Signed-off-by: Chris Friesen <email address hidden>
    Change-Id: I7c6a75ff9bbdddf1bfab3d2ea71a25224293d989

Changed in starlingx:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers