| 2020-05-02 18:08:15 |
Peng Peng |
bug |
|
|
added bug |
| 2020-05-03 14:55:39 |
Peng Peng |
tags |
|
stx.retestneeded |
|
| 2020-05-04 11:31:28 |
Bart Wensley |
bug |
|
|
added subscriber Bart Wensley |
| 2020-05-04 17:54:41 |
Ghada Khalil |
tags |
stx.retestneeded |
stx.retestneeded stx.up |
|
| 2020-05-04 17:56:13 |
Ghada Khalil |
tags |
stx.retestneeded stx.up |
stx.4.0 stx.distcloud stx.retestneeded stx.update |
|
| 2020-05-04 17:56:24 |
Ghada Khalil |
bug |
|
|
added subscriber Daniel Badea |
| 2020-05-04 17:56:29 |
Ghada Khalil |
starlingx: status |
New |
Triaged |
|
| 2020-05-04 17:56:32 |
Ghada Khalil |
starlingx: importance |
Undecided |
Medium |
|
| 2020-05-04 17:56:57 |
Ghada Khalil |
starlingx: assignee |
|
Bart Wensley (bartwensley) |
|
| 2020-05-06 14:58:35 |
Bart Wensley |
starlingx: assignee |
Bart Wensley (bartwensley) |
Don Penney (dpenney) |
|
| 2020-05-06 17:48:21 |
Ghada Khalil |
removed subscriber Daniel Badea |
|
|
|
| 2020-05-08 18:10:31 |
Bart Wensley |
tags |
stx.4.0 stx.distcloud stx.retestneeded stx.update |
stx.4.0 stx.retestneeded stx.update |
|
| 2020-05-28 18:28:03 |
Bill Zvonar |
bug |
|
|
added subscriber Allain Legacy |
| 2020-06-19 20:39:39 |
OpenStack Infra |
starlingx: status |
Triaged |
In Progress |
|
| 2020-06-23 17:26:22 |
OpenStack Infra |
starlingx: status |
In Progress |
Fix Released |
|
| 2020-06-26 13:25:32 |
Peng Peng |
description |
Brief Description
-----------------
With oidc and stx-monitor apps applied on Distributed cloud system, after using patching orch to apply Reboot Request patch on DC, one of SX system patch apply failed by host locked.
Severity
--------
Major
Steps to Reproduce
------------------
applied oidc and stx-monitor app on DC system
apply RR patch on system by using patch strategy
Apply strategy
Expected Behavior
------------------
Patching success on all subcloud
Actual Behavior
----------------
one SX subcloud patching failed
Reproducibility
---------------
Unknown - first time this is seen in sanity, will monitor
System Configuration
--------------------
DC system
Lab-name: WCP_80-91
Branch/Pull Time/Commit
-----------------------
2020-04-29_20-00-00
Last Pass
---------
2020-03-29_16-39-59
Timestamp/Logs
--------------
[sysadmin@controller-0 ~(keystone_admin)]$ system application-list
+--------------------------+---------+-----------------------------------+--------------------------------------+----------+-----------+
| application | version | manifest name | manifest file | status | progress |
+--------------------------+---------+-----------------------------------+--------------------------------------+----------+-----------+
| cert-manager | 1.0-0 | cert-manager-manifest | certmanager-manifest.yaml | applied | completed |
| nginx-ingress-controller | 1.0-0 | nginx-ingress-controller-manifest | nginx_ingress_controller_manifest. | applied | completed |
| | | | yaml | | |
| | | | | | |
| oidc-auth-apps | 1.0-0 | oidc-auth-manifest | manifest.yaml | uploaded | completed |
| platform-integ-apps | 1.0-8 | platform-integration-manifest | manifest.yaml | applied | completed |
| stx-monitor | 1.0-1 | analytics-armada-manifest | wr-analytics.yaml | applied | completed |
+--------------------------+---------+-----------------------------------+--------------------------------------+----------+-----------+
[sysadmin@controller-0 ~(keystone_admin)]$ fm alarm-list
+----------+----------------------------------------------------------------------------------+-------------------+----------+----------------+
| Alarm ID | Reason Text | Entity ID | Severity | Time Stamp |
+----------+----------------------------------------------------------------------------------+-------------------+----------+----------------+
| 400.003 | Evaluation license key will expire on 30-sep-2020; there are 152 days remaining | host=controller-1 | minor | 2020-05-01T16: |
| | in this evaluation | | | 43:53.094570 |
| | | | | |
| 400.003 | Evaluation license key will expire on 30-sep-2020; there are 152 days remaining | host=controller-0 | minor | 2020-05-01T16: |
| | in this evaluation | | | 43:49.649907 |
| | | | | |
| 500.101 | Developer patch certificate is enabled | host=controller | critical | 2020-05-01T00: |
| | | | | 06:02.038877 |
| | | | | |
+----------+----------------------------------------------------------------------------------+-------------------+----------+----------------+
[sysadmin@controller-0 ~(keystone_admin)]$ dcmanager subcloud list
+----+-----------+------------+--------------+---------------+---------+
| id | name | management | availability | deploy status | sync |
+----+-----------+------------+--------------+---------------+---------+
| 2 | subcloud6 | managed | online | complete | in-sync |
| 4 | subcloud4 | managed | online | complete | in-sync |
| 7 | subcloud7 | managed | online | complete | in-sync |
+----+-----------+------------+--------------+---------------+---------+
[sysadmin@controller-0 ~(keystone_admin)]$ sw-patch --os-region-name SystemController upload 2020-04-29_20-00-00_LARGE.patch
2020-04-29_20-00-00_LARGE is now available
[sysadmin@controller-0 ~(keystone_admin)]$ sw-patch --os-region-name SystemController apply 2020-04-29_20-00-00_LARGE
2020-04-29_20-00-00_LARGE is now in the repo
[sysadmin@controller-0 ~(keystone_admin)]$ dcmanager patch-strategy create --subcloud-apply-type parallel --max-parallel-subclouds 10
+------------------------+----------------------------+
| Field | Value |
+------------------------+----------------------------+
| subcloud apply type | parallel |
| max parallel subclouds | 10 |
| stop on failure | False |
| state | initial |
| created_at | 2020-05-02T14:09:14.421615 |
| updated_at | None |
+------------------------+----------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ dcmanager patch-strategy apply
+------------------------+----------------------------+
| Field | Value |
+------------------------+----------------------------+
| subcloud apply type | parallel |
| max parallel subclouds | 10 |
| stop on failure | False |
| state | applying |
| created_at | 2020-05-02T14:09:14.421615 |
| updated_at | 2020-05-02T14:10:19.170945 |
+------------------------+----------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ dcmanager strategy-step list
+------------------+-------+-------------------+---------+----------------------------+-------------+
| cloud | stage | state | details | started_at | finished_at |
+------------------+-------+-------------------+---------+----------------------------+-------------+
| SystemController | 1 | creating strategy | | 2020-05-02 14:10:28.837943 | None |
| subcloud6 | 2 | initial | | None | None |
| subcloud4 | 2 | initial | | None | None |
| subcloud7 | 2 | initial | | None | None |
+------------------+-------+-------------------+---------+----------------------------+-------------+
[sysadmin@controller-0 ~(keystone_admin)]$ dcmanager strategy-step list
+------------------+-------+----------+---------------------------------------------------------------------+----------------------------+----------------------------+
| cloud | stage | state | details | started_at | finished_at |
+------------------+-------+----------+---------------------------------------------------------------------+----------------------------+----------------------------+
| SystemController | 1 | complete | | 2020-05-02 14:10:28.837943 | 2020-05-02 14:55:48.468191 |
| subcloud6 | 2 | failed | Strategy apply failed for subcloud6 - unexpected state abort-failed | 2020-05-02 14:55:58.477515 | 2020-05-02 15:30:22.119729 |
| subcloud4 | 2 | complete | | 2020-05-02 14:55:58.484945 | 2020-05-02 15:40:17.458608 |
| subcloud7 | 2 | complete | | 2020-05-02 14:55:58.495617 | 2020-05-02 15:25:26.728835 |
+------------------+-------+----------+---------------------------------------------------------------------+----------------------------+----------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$
Subcloud6:
[sysadmin@controller-0 ~(keystone_admin)]$ fm alarm-list
+----------+----------------------------------------------------------------------------------+-----------------------+----------+----------------+
| Alarm ID | Reason Text | Entity ID | Severity | Time Stamp |
+----------+----------------------------------------------------------------------------------+-----------------------+----------+----------------+
| 400.001 | Service group controller-services failure; dnsmasq(enabled-active, failed) | service_domain= | critical | 2020-05-02T17: |
| | | controller. | | 43:38.264746 |
| | | service_group= | | |
| | | controller-services. | | |
| | | host=controller-0 | | |
| | | | | |
| 400.002 | Service group controller-services has no active members available; expected 1 | service_domain= | critical | 2020-05-02T15: |
| | active member | controller. | | 03:27.053786 |
| | | service_group= | | |
| | | controller-services | | |
| | | | | |
| 200.001 | controller-0 was administratively locked to take it out-of-service. | host=controller-0 | warning | 2020-05-02T14: |
| | | | | 59:09.530587 |
| | | | | |
| 400.003 | Evaluation license key will expire on 30-sep-2020; there are 151 days remaining | host=controller-0 | minor | 2020-05-02T00: |
| | in this evaluation | | | 59:13.512835 |
| | | | | |
| 500.101 | Developer patch certificate is enabled | host=controller | critical | 2020-05-01T00: |
| | | | | 11:40.719420 |
| | | | | |
+----------+----------------------------------------------------------------------------------+-----------------------+----------+----------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-list
+----+--------------+-------------+----------------+-------------+--------------+
| id | hostname | personality | administrative | operational | availability |
+----+--------------+-------------+----------------+-------------+--------------+
| 1 | controller-0 | controller | locked | disabled | online |
Test Activity
-------------
Regression Testing |
Brief Description
-----------------
With oidc and stx-monitor apps applied on Distributed cloud system, after using patching orch to apply Large patch on DC, one of SX system patch apply failed by host locked.
Severity
--------
Major
Steps to Reproduce
------------------
applied oidc and stx-monitor app on DC system
apply Large patch on system by using patch strategy
Apply strategy
Expected Behavior
------------------
Patching success on all subcloud
Actual Behavior
----------------
one SX subcloud patching failed
Reproducibility
---------------
Unknown - first time this is seen in sanity, will monitor
System Configuration
--------------------
DC system
Lab-name: WCP_80-91
Branch/Pull Time/Commit
-----------------------
2020-04-29_20-00-00
Last Pass
---------
2020-03-29_16-39-59
Timestamp/Logs
--------------
[sysadmin@controller-0 ~(keystone_admin)]$ system application-list
+--------------------------+---------+-----------------------------------+--------------------------------------+----------+-----------+
| application | version | manifest name | manifest file | status | progress |
+--------------------------+---------+-----------------------------------+--------------------------------------+----------+-----------+
| cert-manager | 1.0-0 | cert-manager-manifest | certmanager-manifest.yaml | applied | completed |
| nginx-ingress-controller | 1.0-0 | nginx-ingress-controller-manifest | nginx_ingress_controller_manifest. | applied | completed |
| | | | yaml | | |
| | | | | | |
| oidc-auth-apps | 1.0-0 | oidc-auth-manifest | manifest.yaml | uploaded | completed |
| platform-integ-apps | 1.0-8 | platform-integration-manifest | manifest.yaml | applied | completed |
| stx-monitor | 1.0-1 | analytics-armada-manifest | wr-analytics.yaml | applied | completed |
+--------------------------+---------+-----------------------------------+--------------------------------------+----------+-----------+
[sysadmin@controller-0 ~(keystone_admin)]$ fm alarm-list
+----------+----------------------------------------------------------------------------------+-------------------+----------+----------------+
| Alarm ID | Reason Text | Entity ID | Severity | Time Stamp |
+----------+----------------------------------------------------------------------------------+-------------------+----------+----------------+
| 400.003 | Evaluation license key will expire on 30-sep-2020; there are 152 days remaining | host=controller-1 | minor | 2020-05-01T16: |
| | in this evaluation | | | 43:53.094570 |
| | | | | |
| 400.003 | Evaluation license key will expire on 30-sep-2020; there are 152 days remaining | host=controller-0 | minor | 2020-05-01T16: |
| | in this evaluation | | | 43:49.649907 |
| | | | | |
| 500.101 | Developer patch certificate is enabled | host=controller | critical | 2020-05-01T00: |
| | | | | 06:02.038877 |
| | | | | |
+----------+----------------------------------------------------------------------------------+-------------------+----------+----------------+
[sysadmin@controller-0 ~(keystone_admin)]$ dcmanager subcloud list
+----+-----------+------------+--------------+---------------+---------+
| id | name | management | availability | deploy status | sync |
+----+-----------+------------+--------------+---------------+---------+
| 2 | subcloud6 | managed | online | complete | in-sync |
| 4 | subcloud4 | managed | online | complete | in-sync |
| 7 | subcloud7 | managed | online | complete | in-sync |
+----+-----------+------------+--------------+---------------+---------+
[sysadmin@controller-0 ~(keystone_admin)]$ sw-patch --os-region-name SystemController upload 2020-04-29_20-00-00_LARGE.patch
2020-04-29_20-00-00_LARGE is now available
[sysadmin@controller-0 ~(keystone_admin)]$ sw-patch --os-region-name SystemController apply 2020-04-29_20-00-00_LARGE
2020-04-29_20-00-00_LARGE is now in the repo
[sysadmin@controller-0 ~(keystone_admin)]$ dcmanager patch-strategy create --subcloud-apply-type parallel --max-parallel-subclouds 10
+------------------------+----------------------------+
| Field | Value |
+------------------------+----------------------------+
| subcloud apply type | parallel |
| max parallel subclouds | 10 |
| stop on failure | False |
| state | initial |
| created_at | 2020-05-02T14:09:14.421615 |
| updated_at | None |
+------------------------+----------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ dcmanager patch-strategy apply
+------------------------+----------------------------+
| Field | Value |
+------------------------+----------------------------+
| subcloud apply type | parallel |
| max parallel subclouds | 10 |
| stop on failure | False |
| state | applying |
| created_at | 2020-05-02T14:09:14.421615 |
| updated_at | 2020-05-02T14:10:19.170945 |
+------------------------+----------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ dcmanager strategy-step list
+------------------+-------+-------------------+---------+----------------------------+-------------+
| cloud | stage | state | details | started_at | finished_at |
+------------------+-------+-------------------+---------+----------------------------+-------------+
| SystemController | 1 | creating strategy | | 2020-05-02 14:10:28.837943 | None |
| subcloud6 | 2 | initial | | None | None |
| subcloud4 | 2 | initial | | None | None |
| subcloud7 | 2 | initial | | None | None |
+------------------+-------+-------------------+---------+----------------------------+-------------+
[sysadmin@controller-0 ~(keystone_admin)]$ dcmanager strategy-step list
+------------------+-------+----------+---------------------------------------------------------------------+----------------------------+----------------------------+
| cloud | stage | state | details | started_at | finished_at |
+------------------+-------+----------+---------------------------------------------------------------------+----------------------------+----------------------------+
| SystemController | 1 | complete | | 2020-05-02 14:10:28.837943 | 2020-05-02 14:55:48.468191 |
| subcloud6 | 2 | failed | Strategy apply failed for subcloud6 - unexpected state abort-failed | 2020-05-02 14:55:58.477515 | 2020-05-02 15:30:22.119729 |
| subcloud4 | 2 | complete | | 2020-05-02 14:55:58.484945 | 2020-05-02 15:40:17.458608 |
| subcloud7 | 2 | complete | | 2020-05-02 14:55:58.495617 | 2020-05-02 15:25:26.728835 |
+------------------+-------+----------+---------------------------------------------------------------------+----------------------------+----------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$
Subcloud6:
[sysadmin@controller-0 ~(keystone_admin)]$ fm alarm-list
+----------+----------------------------------------------------------------------------------+-----------------------+----------+----------------+
| Alarm ID | Reason Text | Entity ID | Severity | Time Stamp |
+----------+----------------------------------------------------------------------------------+-----------------------+----------+----------------+
| 400.001 | Service group controller-services failure; dnsmasq(enabled-active, failed) | service_domain= | critical | 2020-05-02T17: |
| | | controller. | | 43:38.264746 |
| | | service_group= | | |
| | | controller-services. | | |
| | | host=controller-0 | | |
| | | | | |
| 400.002 | Service group controller-services has no active members available; expected 1 | service_domain= | critical | 2020-05-02T15: |
| | active member | controller. | | 03:27.053786 |
| | | service_group= | | |
| | | controller-services | | |
| | | | | |
| 200.001 | controller-0 was administratively locked to take it out-of-service. | host=controller-0 | warning | 2020-05-02T14: |
| | | | | 59:09.530587 |
| | | | | |
| 400.003 | Evaluation license key will expire on 30-sep-2020; there are 151 days remaining | host=controller-0 | minor | 2020-05-02T00: |
| | in this evaluation | | | 59:13.512835 |
| | | | | |
| 500.101 | Developer patch certificate is enabled | host=controller | critical | 2020-05-01T00: |
| | | | | 11:40.719420 |
| | | | | |
+----------+----------------------------------------------------------------------------------+-----------------------+----------+----------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-list
+----+--------------+-------------+----------------+-------------+--------------+
| id | hostname | personality | administrative | operational | availability |
+----+--------------+-------------+----------------+-------------+--------------+
| 1 | controller-0 | controller | locked | disabled | online |
Test Activity
-------------
Regression Testing |
|
| 2020-06-26 18:28:21 |
Peng Peng |
tags |
stx.4.0 stx.retestneeded stx.update |
stx.4.0 stx.update |
|
