Kubernetes upgrade failed on subcloud

[S Shatheesh]

Brief Description
-----------------
Kubernetes upgrade failed because Kubelet had to kill KubeApiServer pod due to resource contention.

Steps to Reproduce
------------------
Perform Kubernetes upgrade while kubeapiserver pod is down

Expected Behavior
------------------
Kubernetes upgrade succeeds

Actual Behavior
----------------
Kubernetes upgrade fails

Reproducibility
---------------
Intermittent

System Configuration
--------------------
Subcloud

Test Activity
-------------
Regression Testing

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/nfv/+/910507

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/nfv/+/910508

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/nfv/+/910510

[OpenStack Infra (hudson-openstack)]

Change abandoned by "S Shatheesh <email address hidden>" on branch: master
Review: https://review.opendev.org/c/starlingx/nfv/+/910508
Reason: Created new opendev link with required changes

[OpenStack Infra (hudson-openstack)]

Change abandoned by "S Shatheesh <email address hidden>" on branch: master
Review: https://review.opendev.org/c/starlingx/nfv/+/910507
Reason: Created new opendev link with required changes

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/nfv/+/910510
Committed: https://opendev.org/starlingx/nfv/commit/9c73d3b254ddebd65e6c5d6f7949813c8d08192f
Submitter: "Zuul (22348)"
Branch: master

commit 9c73d3b254ddebd65e6c5d6f7949813c8d08192f
Author: sshathee <email address hidden>
Date: Wed Feb 28 08:55:45 2024 -0500

    Add retry at nfv orchestration level

    This commit introduces retry on failure for cases such
    as kubelet killing pods due to resource contention during
    kubernetes upgrade.

    Test Plan:
        PASS: Simulate kubeapiserver pod failure by adding wrong resource
        in rest api request and check retries.

        PASS: Verify kubernetes orchestrated update works with
        changes on aio-sx

        PASS: Verify changes are working on AIO-DX, with strategy
        created on controller-0 and applied on controller-1

    Closes-Bug: #2053236
    Change-Id: I816b09bb0cd767380e5093d4732d161e4cc8cb24
    Signed-off-by: sshathee <email address hidden>