Wrong version of kubelet after kubernetes upgrade

[Mihnea Saracin]

Brief Description
-----------------
After locking, upgrading kubelet, and then unlocking, kubelet comes up as the old version.

Severity
--------
Critical: System/Feature is not usable due to the defect

Steps to Reproduce
------------------
system kube-upgrade-start v1.19.13
system kube-upgrade-download-images
system kube-upgrade-networking

system host-lock controller-1
system kube-host-upgrade controller-1 kubelet
system host-unlock controller-1

Expected Behavior
------------------
Kubelet has the new version after unlock

Actual Behavior
----------------
Kubelet has the old version after unlock

Reproducibility
---------------
100%

System Configuration
--------------------
Standard system

Branch/Pull Time/Commit
-----------------------
stx master build on 2021-09-10 06:00:51

Last Pass
---------
N/A

Test Activity
-------------
Feature Testing

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/config/+/809158

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/config/+/809158
Committed: https://opendev.org/starlingx/config/commit/02535937a26c7524207468efe66dc52dd72bb95f
Submitter: "Zuul (22348)"
Branch: master

commit 02535937a26c7524207468efe66dc52dd72bb95f
Author: Mihnea Saracin <email address hidden>
Date: Wed Sep 15 12:06:27 2021 +0300

    Use upgraded kubelet on unlock after kubelet upgrade

    When unlocking after kubelet upgrade, there is a problem here:
    https://opendev.org/starlingx/config/src/commit/c784e1ca9a8bd82e86f889b7f621d2363b1393af/sysinv/sysinv/sysinv/sysinv/puppet/kubernetes.py#L383
    the kube_host_upgrade status is no longer equal to
    KUBE_HOST_UPGRADING_KUBELET and the kubelet version is taken from
    the DB which is the old one.
    The kubelet value from the DB is updated when the k8s upgrade is
    complete. (i.e after doing: system kube-upgrade-complete)

    To fix this we added a new state to kube_host_upgrade.status
    defined as kubernetes.KUBE_HOST_UPGRADED_KUBELET. This state will
    indicate that the kubelet was upgraded for a certain host.

    When we complete the upgrade (i.e system kube-upgrade-complete)
    the per-host kube_host_upgrade.status values will be set to None
    so that they're blank in prep for the next k8s upgrade.

    Closes-Bug: 1943690
    Change-Id: Ifdb5c51649831deed76ab9a5370cd293f8a5784a
    Signed-off-by: Mihnea Saracin <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/nfv/+/810014

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/nfv/+/810014
Committed: https://opendev.org/starlingx/nfv/commit/4977e993153e474392156da02d98e5725b8a29f1
Submitter: "Zuul (22348)"
Branch: master

commit 4977e993153e474392156da02d98e5725b8a29f1
Author: albailey <email address hidden>
Date: Mon Sep 20 07:38:08 2021 -0500

    Support for new state for kube host upgrade kubelet

    The kubernetes upgrade procedure in sysinv has changed,
    and as part of the implementation changes, a new
    upgraded-kubelet' state for the hosts structure was added.

    The VIM code has been updated to support that new state.

    Partial-Bug: 1943690
    Depends-On: https://review.opendev.org/c/starlingx/config/+/809158
    Signed-off-by: albailey <email address hidden>
    Change-Id: Ib1163181b19e1032fec83755102d8d556f5b1d06