VIM orchestrator unlocks the host when kubelet upgrade is in progress

[Vanathi Selvaraju]

Brief Description
-----------------
K8s upgrade using VIM orchestrator failed in the host unlock phase as kubelet upgrade was
still ongoing.

Severity
--------

Major

Steps to Reproduce
------------------
sw-manager kube-upgrade-strategy create --to-version 1.22.5
sw-manager kube-upgrade-strategy apply

Expected Behavior
------------------
k8s upgrade should be successful

Actual Behavior
----------------
K8s upgrade failed in host unlock phase.

Reproducibility
---------------
Seen occasionally

System Configuration
--------------------
Standard configuration(with controller-0 , controller-1 and a worker)

Branch/Pull Time/Commit
-----------------------
2023-11-14_22-20-52

Timestamp/Logs
--------------

sysinv 2023-11-17 03:43:17.152 45437 INFO sysinv.api.controllers.v1.host [-] host unlock check didn't pass, so set the ihost_a ction (lock--) back to None and re-raise the exception
sysinv 2023-11-17 03:43:17.166 45437 WARNING wsme.api [-] Client-side error: Can not unlock controller-0 while upgrading kubelet. Wait for kubelet upgrade to complete.: wsme.exc.ClientSideError: Can not unlock controller-0 while upgrading kubelet. Wait for kubelet upgrade to complete.

Test Activity
-------------
System Test

Workaround
----------

Unlock the host(which was failing to unlock) manually.
retrigger k8s upgrade.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/nfv/+/901767

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Vanathi Selvaraju <email address hidden>" on branch: master
Review: https://review.opendev.org/c/starlingx/nfv/+/901767
Reason: duplicate

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/nfv/+/901768

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Vanathi Selvaraju <email address hidden>" on branch: master
Review: https://review.opendev.org/c/starlingx/nfv/+/901768
Reason: duplicate

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Vanathi Selvaraju <email address hidden>" on branch: master
Review: https://review.opendev.org/c/starlingx/nfv/+/901638
Reason: duplicate

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/nfv/+/901638
Committed: https://opendev.org/starlingx/nfv/commit/71406be649e3dea25c0bd88ae187cd9650d15e79
Submitter: "Zuul (22348)"
Branch: master

commit 71406be649e3dea25c0bd88ae187cd9650d15e79
Author: Vanathi.Selvaraju <email address hidden>
Date: Tue Nov 21 21:10:52 2023 -0500

    K8s upgrade failed in host-unlock phase

    K8s upgrade failed in the host-unlock phase
    as kubelet upgrade was ongoing, VIM tries
    to unlock the host before completion of
    kubelet upgrade.

    Test Plan:
    PASSED: On a DX system, apply the fix and trigger
    K8s upgrade.
    PASSED: On a SX system, do three consecutive k8s upgrade.
    PASSED: On DX system ,two consecutive k8s upgrade.

    Closes-Bug: 2044209

    Change-Id: If4bcabcba6aabd1aee6918d73fd6994e94f0b1f8
    Signed-off-by: Vanathi.Selvaraju <email address hidden>