robustness: error in k8s upgrade ends up stuck in upgrading-second-master
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
StarlingX |
Fix Released
|
Medium
|
sachin |
Bug Description
While testing a bug in k8s upgrades which led to a late-stage failure upgrading the second controller node, the "sudo kubeadm upgrade node" command failed right near the end after the new /var/lib/
Each of the above issues on their own would mean that we can't retry the "system kube-host-upgrade controller-X control-plane" command as it fails the pre-checks.
Severity
Major
Steps to Reproduce
Force a late-stage failure of the "kubeadm upgrade node" command when doing a kubernetes upgrade on a control plane node.
Expected Behavior
System transitions to state of "upgrading-
Actual Behavior
System stays at "upgrading-
Reproducibility
Reproducible (with a late-state failure of the puppet manifest).
System Configuration
Multi-node standard lab. Might also show up in AIO-SX if there is a failure in the k8s control plane upgrade puppet manifest.
Changed in starlingx: | |
assignee: | nobody → sachin (skrishn5) |
tags: | added: stx.containers stx.update |
Changed in starlingx: | |
importance: | Undecided → Medium |
tags: | added: stx.8.0 |
Fix proposed to branch: master /review. opendev. org/c/starlingx /config/ +/867255
Review: https:/