Comment 0 for bug 2038341

Brief Description:

K8s upgrade-abort in a SX stuck at upgrade-aborting state

Severity:
Major

Steps to Reproduce:

system kube-upgrade-start v1.22.5
system kube-upgrade-download-images
system kube-upgrade-networking
system kube-host-upgrade controller-0 control-plane
system kube-upgrade-abort

After controller-0 is upgraded abort the upgrade

Expected Behavior:

Upgrade should be in upgrade-aborted state

Actual Behavior:

sysadmin@controller-0 (keystone_admin)]$ system kube-upgrade-show
-----------{}{}-----------------------------------+

Property Value
-----------{}{}-----------------------------------+

uuid aa3bf9f5-e677-4e09-8da8-fd1fad29bd24
from_version v1.21.8
to_version v1.22.5
state upgrade-aborting
created_at 2023-09-29T01:52:32.964855+00:00
updated_at 2023-09-29T02:02:57.925764+00:00
-----------{}{}-----------------------------------+

Reproducibility:
100%

System Configuration

AIO-SX

Last Pass

NA

Timestamp/Logs

sysinv 2023-09-29 02:06:15.360 20601 WARNING sysinv.puppet.kubernetes [-] Exception generating bootstrap token: subprocess.CalledProcessError: Command '['/usr/local/kubernetes/1.21.8/stage1/usr/bin/kubeadm', 'init', 'phase', 'upload-certs', '--upload-certs', '--config', '/tmp/tmp4c83zp55.yaml']' returned non-zero exit status 1.
sysinv 2023-09-29 02:06:15.368 20601 WARNING sysinv.puppet.kubernetes [-] Retrying in _get_kubernetes_join_cmd
sysinv 2023-09-29 02:06:15.368 20601 ERROR sysinv.conductor.manager [-] _config_update_puppet Failed to generate bootstrap token: sysinv.common.exception.SysinvException: Failed to generate bootstrap token

Alarms

NA

Test Activity

Feature Testing

Workaround

Describe workaround if available