SX K8s upgrade abort stuck at upgrade-aborting

[Boovan Rajendran]

Brief Description:

K8s upgrade-abort in a SX stuck at upgrade-aborting state

Severity:
Major

Steps to Reproduce:

system kube-upgrade-start v1.22.5
system kube-upgrade-download-images
system kube-upgrade-networking
system kube-host-upgrade controller-0 control-plane
system kube-upgrade-abort

After controller-0 is upgraded abort the upgrade

Expected Behavior:

Upgrade should be in upgrade-aborted state

Actual Behavior:

[sysadmin@controller-0 ~(keystone_admin)]$ system kube-upgrade-abort
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | 527d95e1-307c-4836-9614-f8ebd844691f |
| from_version | v1.21.8 |
| to_version | v1.22.5 |
| state | upgrade-aborting |
| created_at | 2023-10-03T07:05:11.711079+00:00 |
| updated_at | 2023-10-03T07:29:36.126765+00:00 |
+--------------+--------------------------------------+

Reproducibility:
100%

System Configuration

AIO-SX

Last Pass

NA

Timestamp/Logs

sysinv 2023-09-29 02:06:15.360 20601 WARNING sysinv.puppet.kubernetes [-] Exception generating bootstrap token: subprocess.CalledProcessError: Command '['/usr/local/kubernetes/1.21.8/stage1/usr/bin/kubeadm', 'init', 'phase', 'upload-certs', '--upload-certs', '--config', '/tmp/tmp4c83zp55.yaml']' returned non-zero exit status 1.
sysinv 2023-09-29 02:06:15.368 20601 WARNING sysinv.puppet.kubernetes [-] Retrying in _get_kubernetes_join_cmd
sysinv 2023-09-29 02:06:15.368 20601 ERROR sysinv.conductor.manager [-] _config_update_puppet Failed to generate bootstrap token: sysinv.common.exception.SysinvException: Failed to generate bootstrap token

Alarms

NA

Test Activity

Feature Testing

Workaround

Describe workaround if available

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/config/+/897230

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/config/+/897230
Committed: https://opendev.org/starlingx/config/commit/03eb0e8eeebc380a7ce019e2083db8b6ffe379c4
Submitter: "Zuul (22348)"
Branch: master

commit 03eb0e8eeebc380a7ce019e2083db8b6ffe379c4
Author: Boovan Rajendran <email address hidden>
Date: Tue Oct 3 10:05:45 2023 -0400

    Replace k8S API references of v1beta3 to v1beta2

    Abort operation is failing when execute the 'kube-upgrade-abort'
    system command during k8s upgrade from 1.21.8 to 1.22.5 after
    upgrading the control plane.

    The kubeadm-config ClusterConfiguration is updated with k8s API
    reference 'kubeadm.k8s.io/v1beta3' during K8s upgrade from 1.21.8 to
    1.22.5. The 'kubeadm.k8s.io/v1beta3' only supports from K8s 1.22.x.
    This should revert back to 'kubeadm.k8s.io/v1beta2' during the
    abort operation.

    This change replaces k8s API references of v1beta3 to v1beta2
    when abort during k8s upgrade from 1.21.8 to 1.22.5.

    TEST PLAN:
    PASS: Fresh install ISO as AIO-SX, perform k8s upgrade from 1.21 to
    1.22 and check by running 'system kube-upgrade-abort' command after
    upgrading control plane from k8s 1.21.8 to 1.22.5 and verify upgrade
    aborted successfully.

    Closes-Bug: 2038341

    Change-Id: Ibeafe4a15e88cc211a747e90b05ef652aa3b168b
    Signed-off-by: Boovan Rajendran <email address hidden>