DC upgrade orchestration trying to remove patch that is not applied to central cloud

[Christopher de Oliveira Souza]

Brief Description
----------------------
During the subcloud upgrade orchestration It was observed that upgrade orchestration was trying to remove the 21.12 patch from subcloud. This patch was applied to subcloud after the central cloud was upgraded from 21.12 to 22.12. The patch it was trying to remove was not installed in Central cloud.

Severity
---------------------
Major:

Steps to Reproduce
---------------------
1. Central cloud upgraded from 21.12 to 22.12
2. Subcloud 3 no reboot patch applied as below. To have a fix for subcloud upgrade.
3. Upgrade orchestration was executed from the central cloud to upgrade subcloud.
As soon after upgrade orchestration started sw-patch query have below. Patch was in partial remove.

Expected Behavior
---------------------
Upgrade orchestration shouldn't try to remove patches from subcloud

Actual Behavior
---------------------
Patch is being partially removed and the upgrade start failed due to an alarm.

Reproducibility
---------------------
100% reproduceable

System Configuration
---------------------
DC with AIO-Dx subcloud

Last Pass
---------------------
Not known

Timestamp/Logs
---------------------
3-03-30T15:11:04: sw-patch-agent[98692]: patch_agent.py(559): INFO: To install: {}
2023-03-30T15:11:04: sw-patch-agent[98692]: patch_agent.py(560): INFO: To remove: []
2023-03-30T15:11:04: sw-patch-agent[98692]: patch_agent.py(561): INFO: Missing: []
2023-03-30T15:11:04: sw-patch-controller-daemon[98734]: patch_controller.py(146): INFO: Agent controller-0 (fd00:8:19::3) reporting out_of_date=False, requires_reboot=False
2023-03-30T15:11:04: sw-patch-agent[98692]: patch_agent.py(189): INFO: Handling detailed query
2023-03-30T15:11:04: sw-patch-agent[98692]: patch_agent.py(189): INFO: Handling detailed query
2023-03-30T15:12:38: sw-patch-controller-daemon[98734]: patch_controller.py(1250): INFO: Removing patches: CGTS-45775
2023-03-30T15:12:38: sw-patch-controller-daemon[98734]: patch_controller.py(1342): INFO: Removing patch: CGTS-45775
2023-03-30T15:12:38: sw-patch-controller-daemon[98734]: patch_controller.py(1405): INFO: Repo[22.12] updated:
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete

2023-03-30T15:12:44: sw-patch-controller-daemon[98734]: patch_controller.py(1405): INFO: Repo[21.12] updated:
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete

2023-03-30T15:12:44: sw-patch-controller-daemon[98734]: patch_controller.py(298): INFO: sending sync req
2023-03-30T15:13:01: sw-patch-controller-daemon[98734]: patch_controller.py(146): INFO: Agent controller-1 (fd00:8:19::4) reporting out_of_date=True, requires_reboot=False
2023-03-30T15:13:01: sw-patch-controller-daemon[98734]: patch_controller.py(146): INFO: Agent compute-0 (fd00:8:19::d4c0) reporting out_of_date=True, requires_reboot=False
2023-03-30T15:13:01: sw-patch-agent[98692]: patch_agent.py(557): INFO: Patch state query returns True
2023-03-30T15:13:01: sw-patch-agent[98692]: patch_agent.py(558): INFO: Installed: {u'sysinv': u'1.0-2541.tis@x86_64'}
2023-03-30T15:13:01: sw-patch-agent[98692]: patch_agent.py(559): INFO: To install: {u'sysinv': u'1.0-2539.tis@x86_64'}
2023-03-30T15:13:01: sw-patch-agent[98692]: patch_agent.py(560): INFO: To remove: []
2023-03-30T15:13:01: sw-patch-agent[98692]: patch_agent.py(561): INFO: Missing: []
2023-03-30T15:13:01: sw-patch-controller-daemon[98734]: patch_controller.py(146): INFO: Agent controller-0 (fd00:8:19::3) reporting out_of_date=True, requires_reboot=False
2023-03-30T15:13:0

Alarms
---------------------
[sysadmin@controller-0 ~(keystone_admin)]$ fm alarm-list
-----------------------------------------------------------------------------------------------------------------------------
| Alarm ID | Reason Text | Entity ID | Severity | Time Stamp |
-----------------------------------------------------------------------------------------------------------------------------
| 900.001 | Patching operation in progress | host=controller | minor | 2023-03-30T15:13: |
| | | | | 05.153552 |
| | | | | |
| 100.104 | File System threshold exceeded ; threshold 80.00%, actual 82.80% | host=controller-0. | major | 2023-03-30T15:11: |
| | | filesystem=/ | | 11.906193 |
| | | | | |
| 500.101 | Developer patch certificate is enabled | host=controller | critical | 2023-03-30T03:05: |
| | | | | 34.689599 |
| | | | | |
-----------------------------------------------------------------------------------------------------------------------------

Test Activity
--------------------------
Regression

Workaround
--------------------------
NA

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/distcloud/+/880673

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/distcloud/+/880673
Committed: https://opendev.org/starlingx/distcloud/commit/5237ab5858d6d7d89d5fc6864dc94da59641ade3
Submitter: "Zuul (22348)"
Branch: master

commit 5237ab5858d6d7d89d5fc6864dc94da59641ade3
Author: Christopher Souza <email address hidden>
Date: Mon Apr 17 13:31:15 2023 -0300

    update upgrade orch to not remove applied patches

    In this commit, the DC upgrade orchestrator was updated to not remove
    applied patches from the subcloud when the patch is not applied
    on system controller.

    Test Plan:
    PASS: Upgrade system controller to 22.12, then apply a patch
    on the subcloud and run the upgrade orchestration and verify
    that the orchestration finished successfully.
    PASS: Apply 21.12 patch on system controller, then commit the patch
    and run patch orchestration. Then upgrade the system controller and
    the subcloud and verify that the orchestration finished successfully.
    PASS: Apply 21.12 patch on the system controller, then run patch
    orchestration and upgrade the system controller and the subcloud
    and verify that the orchestration finished successfully.
    PASS: Upgrade system controller to 22.12, then apply a patch on
    system controller and upgrade the subcloud and verify that
    the orchestration reached the starting upgrade stage.

    Closes-Bug: 2016701

    Signed-off-by: Christopher Souza <email address hidden>
    Change-Id: Id914413c99a46782170300715b78d3129bd46720