Comment 0 for bug 1856226

Brief Description
-----------------
Applying reboot required patched to DC, three subcloud apply success, but one subcloud appl failed. 900.103 alarm "Software patch auto-apply failed" raised

Severity
--------
Major

Steps to Reproduce
------------------
1. apply/upload PATCH.ENABLE_DEV_CERTIFICATE-19.12 to system controller
2. dcmanager patch-strategy create --subcloud-apply-type parallel --max-parallel-subclouds 10
3. dcmanager patch-strategy apply
4. after patch apply success, dcmanager patch-strategy delete
5. upload/apply 2019-12-08_20-00-00_RR_ALLNODES.patch to system controller
6. dcmanager patch-strategy create --subcloud-apply-type parallel --max-parallel-subclouds 10
7. dcmanager patch-strategy apply
8. dcmanager strategy-step list

TC-name: RR patching test

Expected Behavior
------------------
all subcloud patching apply success

Actual Behavior
----------------
subclou1 patching failed

Reproducibility
---------------
Unknown - first time this is seen in sanity, will monitor

System Configuration
--------------------
DC system
IPv6

Lab-name: DC

Branch/Pull Time/Commit
-----------------------
2019-12-08_20-00-00

Last Pass
---------
unknown

Timestamp/Logs
--------------
[sysadmin@controller-0 ~(keystone_admin)]$ dcmanager strategy-step list
+------------------+-------+----------+----------------------------------------------------------------+----------------------------+----------------------------+
| cloud | stage | state | details | started_at | finished_at |
+------------------+-------+----------+----------------------------------------------------------------+----------------------------+----------------------------+
| SystemController | 1 | complete | | 2019-12-12 16:08:43.254742 | 2019-12-12 16:37:27.462675 |
| subcloud1 | 2 | failed | Strategy apply failed for subcloud1 - unexpected state aborted | 2019-12-12 16:37:37.469304 | 2019-12-12 16:41:31.093920 |
| subcloud4 | 2 | complete | | 2019-12-12 16:37:37.480419 | 2019-12-12 17:04:42.760715 |
| subcloud5 | 2 | complete | | 2019-12-12 16:37:37.494256 | 2019-12-12 17:05:52.441619 |
| subcloud6 | 2 | complete | | 2019-12-12 16:37:37.510882 | 2019-12-12 16:50:22.378693 |
+------------------+-------+----------+----------------------------------------------------------------+----------------------------+----------------------------+

Subclud1:
[sysadmin@controller-0 ~(keystone_admin)]$ fm alarm-list
+----------+----------------------------------------+------------------------+----------+-------------------+
| Alarm ID | Reason Text | Entity ID | Severity | Time Stamp |
+----------+----------------------------------------+------------------------+----------+-------------------+
| 900.103 | Software patch auto-apply failed | orchestration=sw-patch | critical | 2019-12-12T16:40: |
| | | | | 51.149126 |
| | | | | |
| 900.001 | Patching operation in progress | host=controller | minor | 2019-12-12T16:37: |
| | | | | 54.527125 |
| | | | | |
| 500.101 | Developer patch certificate is enabled | host=controller | critical | 2019-12-12T15:53: |
| | | | | 35.613420 |
| | | | | |
+----------+----------------------------------------+------------------------+----------+-------------------+

Test Activity
-------------
Regression Testing