Debian: no reboot Patch installs on standby controller despite failure in restart script

[Jessica Castelino]

Brief Description
-----------------
Tried to install an in-service patch on a system which had a faulty restart script. The installation fails on controller-0 but it is successful on controller-1.

Severity
-----------------
standard

Steps to Reproduce
-----------------
1. Uploaded and applied RESTART_FAILURE_INSVC.patch
Used patch strategy (eg. from horizon) to create a strategy and apply it

2. This patch installed on the standby controller, then failed install on the active controller

Expected Behavior
-----------------
The patch should not have failed install ie. should not have installed on the standby controller

Actual Behavior
-----------------
In step 2 the patch installed on the standby controller then continued on to try to install on the active controller where it then failed

Once in failed state, try to remove the patch as follows:

Select Patch Remove then run Patch Install on the standby then the active controller.

That completed but the active controller was still in Reboot required state ie. requires_reboot is still True

Workaround: Had to swact, then lock and unlock to clear the Reboot required state at this point.

Reproducibility
-----------------
yes

System Configuration
-----------------
STD with worker

Load info
-----------------
2022-12-13_22-00-16

Last Pass
-----------------
new test in Debian

Timestamp/Logs
-----------------
N/A

Alarms
-----------------
N/A

Test Activity
-----------------
Feature Testing, Regression Testing

Workaround
-----------------
N/A

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/update/+/867930

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/update/+/867930
Committed: https://opendev.org/starlingx/update/commit/8b599699ea77971d41c9949d2143d4f93c1ecc95
Submitter: "Zuul (22348)"
Branch: master

commit 8b599699ea77971d41c9949d2143d4f93c1ecc95
Author: Jessica Castelino <email address hidden>
Date: Fri Dec 16 02:59:19 2022 +0000

    Sync patch restart scripts to other hosts

    This commit ensures that the patch restart scripts for in-service
    patching are copied to all hosts. Subsequently, we have moved the
    in-service patch restart script staging directory from
    /run/patching/patch-scripts to /var/www/pages/updates/patch-scripts
    which can be rsynced.

    Test Plan:
    [PASS] Verify from patching-insvc.log that the restart scripts
           are run on all hosts

    Closes-Bug: 1999841
    Signed-off-by: Jessica Castelino <email address hidden>
    Change-Id: I25f9b13777a9fc4ff7ee5c15d38ceb80dbfffcf1