Distributed Cloud Ipv6: subcloud patch auto-apply failed

Bug #1856226 reported by Peng Peng on 2019-12-12
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Medium
Al Bailey

Bug Description

Brief Description
-----------------
Applying reboot required patched to DC, three subcloud apply success, but one subcloud appl failed. 900.103 alarm "Software patch auto-apply failed" raised

Severity
--------
Major

Steps to Reproduce
------------------
1. apply/upload PATCH.ENABLE_DEV_CERTIFICATE-19.12 to system controller
2. dcmanager patch-strategy create --subcloud-apply-type parallel --max-parallel-subclouds 10
3. dcmanager patch-strategy apply
4. after patch apply success, dcmanager patch-strategy delete
5. upload/apply 2019-12-08_20-00-00_RR_ALLNODES.patch to system controller
6. dcmanager patch-strategy create --subcloud-apply-type parallel --max-parallel-subclouds 10
7. dcmanager patch-strategy apply
8. dcmanager strategy-step list

TC-name: RR patching test

Expected Behavior
------------------
all subcloud patching apply success

Actual Behavior
----------------
subclou1 patching failed

Reproducibility
---------------
Unknown - first time this is seen in this load

System Configuration
--------------------
DC system
IPv6

Lab-name: DC

Branch/Pull Time/Commit
-----------------------
2019-12-08_20-00-00

Last Pass
---------
2019-10-06 load

Timestamp/Logs
--------------
[sysadmin@controller-0 ~(keystone_admin)]$ dcmanager strategy-step list
+------------------+-------+----------+----------------------------------------------------------------+----------------------------+----------------------------+
| cloud | stage | state | details | started_at | finished_at |
+------------------+-------+----------+----------------------------------------------------------------+----------------------------+----------------------------+
| SystemController | 1 | complete | | 2019-12-12 16:08:43.254742 | 2019-12-12 16:37:27.462675 |
| subcloud1 | 2 | failed | Strategy apply failed for subcloud1 - unexpected state aborted | 2019-12-12 16:37:37.469304 | 2019-12-12 16:41:31.093920 |
| subcloud4 | 2 | complete | | 2019-12-12 16:37:37.480419 | 2019-12-12 17:04:42.760715 |
| subcloud5 | 2 | complete | | 2019-12-12 16:37:37.494256 | 2019-12-12 17:05:52.441619 |
| subcloud6 | 2 | complete | | 2019-12-12 16:37:37.510882 | 2019-12-12 16:50:22.378693 |
+------------------+-------+----------+----------------------------------------------------------------+----------------------------+----------------------------+

Subclud1:
[sysadmin@controller-0 ~(keystone_admin)]$ fm alarm-list
+----------+----------------------------------------+------------------------+----------+-------------------+
| Alarm ID | Reason Text | Entity ID | Severity | Time Stamp |
+----------+----------------------------------------+------------------------+----------+-------------------+
| 900.103 | Software patch auto-apply failed | orchestration=sw-patch | critical | 2019-12-12T16:40: |
| | | | | 51.149126 |
| | | | | |
| 900.001 | Patching operation in progress | host=controller | minor | 2019-12-12T16:37: |
| | | | | 54.527125 |
| | | | | |
| 500.101 | Developer patch certificate is enabled | host=controller | critical | 2019-12-12T15:53: |
| | | | | 35.613420 |
| | | | | |
+----------+----------------------------------------+------------------------+----------+-------------------+

Test Activity
-------------
Regression Testing

Peng Peng (ppeng) wrote :
Peng Peng (ppeng) wrote :
Ghada Khalil (gkhalil) on 2019-12-12
tags: added: stx.distcloud
Yang Liu (yliu12) on 2019-12-12
description: updated
tags: added: stx.retestneeded
Ghada Khalil (gkhalil) wrote :

stx.4.0 / medium priority - Affects a specific config. This appears to only be affecting standard subclouds only. AIO-DX & AIO-SX subclouds don't have this issue.

tags: added: stx.4.0
Changed in starlingx:
importance: Undecided → Medium
status: New → Triaged
assignee: nobody → Al Bailey (albailey1974)
Peng Peng (ppeng) wrote :

2019-12-13_19-03-42_RR_ALLNODES.patch
BUILD_ID="2019-12-13_19-03-42"

passed, included the same subcloud1

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers