Fix timezone issue in _audit_kube_upgrade_states periodic task

[sachin]

Brief Description

While testing kube upgrade [state transition|https://jira.wrs.com/browse/CGTS-40562 bug in k8s upgrades which led to a failure, the "_audit_kube_upgrade_states" perodic task created below stack trace

sysinv 2023-02-24 13:58:58.239 333768 ERROR sysinv.openstack.common.periodic_task [-] Error during ConductorManager._audit_kube_upgrade_states: can't subtract offset-naive and offset-aware datetimes: TypeError: can't subtract offset-naive and offset-aware datetimes 2023-02-24 13:58:58.239 333768 ERROR sysinv.openstack.common.periodic_task Traceback (most recent call last): 2023-02-24 13:58:58.239 333768 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python3/dist-packages/sysinv/openstack/common/periodic_task.py", line 180, in run_periodic_tasks 2023-02-24 13:58:58.239 333768 ERROR sysinv.openstack.common.periodic_task task(self, context) 2023-02-24 13:58:58.239 333768 ERROR sysinv.openstack.common.periodic_task File "/usr/lib/python3/dist-packages/sysinv/conductor/manager.py", line 16639, in _audit_kube_upgrade_states 2023-02-24 13:58:58.239 333768 ERROR sysinv.openstack.common.periodic_task if datetime.utcnow() - kube_upgrade_time_stamp >= timedelta( 2023-02-24 13:58:58.239 333768 ERROR sysinv.openstack.common.periodic_task TypeError: can't subtract offset-naive and offset-aware datetimes 2023-02-24 13:58:58.239 333768 ERROR sysinv.openstack.common.periodic_task
Severity

Major

Steps to Reproduce

After kube upgrade status is set to "upgraded-second-master" manually update the sysinv db and set kube upgrade status to "upgrading-second-master".

Expected Behavior

_audit_kube_upgrade_states periodic task will transitions to the state of "upgrading-second-master-failed" after two cycles.

Actual Behavior

_audit_kube_upgrade_states errors out and the kube upgrade status will remain as "upgrading-second-master".

Reproducibility

Reproducible.

Last Pass

Unknown

Timestamp/Logs

N/A

Alarms

No

Test Activity

Testing K8s upgrade.

Workaround

Manually set the kube upgrade status to '*-failed" by executing "system kube-upgrade-failed"

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/config/+/875501
Committed: https://opendev.org/starlingx/config/commit/d6c38e9a50c440a296e0dc052554f6740c3e7480
Submitter: "Zuul (22348)"
Branch: master

commit d6c38e9a50c440a296e0dc052554f6740c3e7480
Author: Sachin Gopala Krishna <email address hidden>
Date: Mon Feb 27 10:00:34 2023 -0500

    Fix timezone exception during kube_upgrade_state periodic audit

    periodic task _audit_kube_upgrade_states to
    automatically change the kube_upgrade state to *-failed
    if the specific state is stuck 'upgrading-*' for more
    than 2 cycles of _audit_kube_upgrade_states. This fixes
    an exception in the calculation of timedelta when there
    is timezone mismatch between datetime object (from the
    sysinv database) and current offset-naive.

    Test Plan:
    PASS: Manually edit kube_upgrade state to upgrading-* after kube_upgrade
    completion and wait for one hour and verify state transition to *-failed
    based on updated_at time stamp

    Closes-Bug: 2008889

    Signed-off-by: Sachin Gopala Krishna <email address hidden>
    Change-Id: Idbecf3d83f5de3ff2da7e30b5a63220105cc0bbf