IPv6 Distributed Cloud: INSVC_ALLNODES not reboot request test patch Strategy apply failed

[Peng Peng]

Brief Description
-----------------
After PATCH.ENABLE_DEV_CERTIFICATE-20.01.patch applied to the whole system, trying to apply 2020-01-28_07-44-47_INSVC_ALLNODES.patch to system. One subcloud apply failed by "Strategy apply failed for subcloud1 - unexpected state aborted"

Severity
--------
Major

Steps to Reproduce
------------------
sw-patch --os-region-name SystemController upload 2020-01-28_07-44-47_INSVC_ALLNODES.patch
sw-patch query
sw-patch --os-region-name SystemController apply 2020-01-28_07-44-47_INSVC_ALLNODES

dcmanager patch-strategy create --subcloud-apply-type parallel --max-parallel-subclouds 10

# for SX subcloud
dcmanager patch-strategy-config update --storage-apply-type parallel --default-instance-action stop-start subcloud6 --alarm-restriction-type relaxed --max-parallel-workers 10 --worker-apply-type parallel

dcmanager patch-strategy apply

dcmanager strategy-step list

Expected Behavior
------------------
strategy apply success on all subcloud

Actual Behavior
----------------
one subcloud (regular system) strategy apply failed

Reproducibility
---------------
Intermittent
(1/10)

System Configuration
--------------------
DC system

Lab-name: WCP_80-91

Branch/Pull Time/Commit
-----------------------
2020-01-28_18-49-15

Last Pass
---------
2020-01-24_18-07-30

Timestamp/Logs
--------------
[sysadmin@controller-0 ~(keystone_admin)]$ dcmanager strategy-step list
+------------------+-------+----------+----------------------------------------------------------------+----------------------------+----------------------------+
| cloud | stage | state | details | started_at | finished_at |
+------------------+-------+----------+----------------------------------------------------------------+----------------------------+----------------------------+
| SystemController | 1 | complete | | 2020-01-29 20:14:57.030274 | 2020-01-29 20:18:37.246294 |
| subcloud4 | 2 | complete | | 2020-01-29 20:18:47.254775 | 2020-01-29 20:22:38.657157 |
| subcloud1 | 2 | failed | Strategy apply failed for subcloud1 - unexpected state aborted | 2020-01-29 20:18:47.260621 | 2020-01-29 20:21:38.621759 |
| subcloud6 | 2 | complete | | 2020-01-29 20:18:47.270186 | 2020-01-29 20:21:38.660493 |
| subcloud5 | 2 | complete | | 2020-01-29 20:18:47.279324 | 2020-01-29 20:21:38.955654 |
| subcloud7 | 2 | complete | | 2020-01-29 20:18:47.288403 | 2020-01-29 20:21:48.842969 |
+------------------+-------+----------+----------------------------------------------------------------+----------------------------+----------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$

Test Activity
-------------
Sanity

[Peng Peng (ppeng)]

log @
https://files.starlingx.kube.cengn.ca/launchpad/1861336

[Ghada Khalil (gkhalil)]

stx.4.0 / medium priority -- appears to be a very intermittent issue, but should be investigated by dev prime

[Dariush Eslimi (deslimi)]

There has been many improvement in this area, please retest.

[Peng Peng (ppeng)]

Verified on
DC-3

[sysadmin@controller-1 ~(keystone_admin)]$ sw-patch query
             Patch ID RR Release Patch State
================================== == ======= ===========
2020-03-25_21-02-05_INSVC_ALLNODES N 20.03 Applied
2020-03-25_21-02-05_RR_ALLNODES Y 20.03 Applied
PATCH.ENABLE_DEV_CERTIFICATE-20.03 N 20.03 Applied

[sysadmin@controller-1 ~(keystone_admin)]$ cat /etc/build.info

BUILD_ID="2020-03-25_21-02-05"