| 2019-12-11 15:46:57 |
Peng Peng |
bug |
|
|
added bug |
| 2019-12-11 15:48:04 |
Peng Peng |
attachment added |
|
ALL_NODES_20191211.152313.tar https://bugs.launchpad.net/starlingx/+bug/1856064/+attachment/5311791/+files/ALL_NODES_20191211.152313.tar |
|
| 2019-12-11 17:26:59 |
Yang Liu |
description |
Brief Description
-----------------
After lock/unlock one compute node, the active controller became degraded. 200.006 alarm raised.
After active controller force reboot, the system was recovered and alarm was cleared.
Severity
--------
Major
Steps to Reproduce
------------------
as description
TC-name: mtc/test_lock_unlock_host.py::test_lock_unlock_host[compute]
Expected Behavior
------------------
Actual Behavior
----------------
Reproducibility
---------------
Unknown - first time this is seen in sanity, will monitor
System Configuration
--------------------
Multi-node system
IPv4
Lab-name: WCP_3-6
Branch/Pull Time/Commit
-----------------------
2019-12-10_20-00-00
Last Pass
---------
2019-12-10_20-00-00 on (WP_8-12)
Timestamp/Logs
--------------
[2019-12-11 08:58:20,124] 311 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.1:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-list'
[2019-12-11 08:58:21,300] 433 DEBUG MainThread ssh.expect :: Output:
+----+--------------+-------------+----------------+-------------+--------------+
| id | hostname | personality | administrative | operational | availability |
+----+--------------+-------------+----------------+-------------+--------------+
| 1 | controller-0 | controller | unlocked | enabled | available |
| 2 | compute-0 | worker | unlocked | enabled | available |
| 3 | compute-1 | worker | unlocked | enabled | available |
| 4 | controller-1 | controller | unlocked | enabled | available |
+----+--------------+-------------+----------------+-------------+--------------+
[2019-12-11 08:58:22,661] 311 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.1:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-lock compute-0'
[2019-12-11 08:59:40,320] 311 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.1:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-unlock compute-0'
[2019-12-11 09:05:59,264] 311 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.1:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-list'
[2019-12-11 09:06:00,442] 433 DEBUG MainThread ssh.expect :: Output:
+----+--------------+-------------+----------------+-------------+--------------+
| id | hostname | personality | administrative | operational | availability |
+----+--------------+-------------+----------------+-------------+--------------+
| 1 | controller-0 | controller | unlocked | enabled | degraded |
| 2 | compute-0 | worker | unlocked | enabled | available |
| 3 | compute-1 | worker | unlocked | enabled | available |
| 4 | controller-1 | controller | unlocked | enabled | available |
+----+--------------+-------------+----------------+-------------+--------------+
[sysadmin@controller-0 ~(keystone_admin)]$
[2019-12-11 09:11:08,717] 311 DEBUG MainThread ssh.send :: Send 'fm --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.1:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne alarm-list --nowrap --uuid'
[2019-12-11 09:11:09,693] 433 DEBUG MainThread ssh.expect :: Output:
+--------------------------------------+----------+------------------------------------------------------------------------------------------------------------------------+--------------------------------+----------+----------------------------+
| UUID | Alarm ID | Reason Text | Entity ID | Severity | Time Stamp |
+--------------------------------------+----------+------------------------------------------------------------------------------------------------------------------------+--------------------------------+----------+----------------------------+
| 26e10dab-15dd-45ee-b5ac-4ae73bb5db8d | 200.006 | controller-0 is degraded due to the failure of its 'ceph' process. Auto recovery of this major process is in progress. | host=controller-0.process=ceph | major | 2019-12-11T09:00:12.697608 |
+--------------------------------------+----------+------------------------------------------------------------------------------------------------------------------------+--------------------------------+----------+----------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$
controller-0:~$
[2019-12-11 09:16:45,554] 166 INFO MainThread host_helper.reboot_hosts:: Rebooting controller-0
[2019-12-11 09:16:45,554] 311 DEBUG MainThread ssh.send :: Send 'sudo reboot -f'
Test Activity
-------------
Sanity |
Brief Description
-----------------
After lock/unlock one compute node, the active controller became degraded. 200.006 alarm raised.
After active controller force reboot, the system was recovered and alarm was cleared.
Severity
--------
Major
Steps to Reproduce
------------------
as description
TC-name: mtc/test_lock_unlock_host.py::test_lock_unlock_host[compute]
Expected Behavior
------------------
Actual Behavior
----------------
Reproducibility
---------------
Unknown - first time this is seen in sanity, will monitor
System Configuration
--------------------
Multi-node system
IPv4
Lab-name: WCP_3-6
Branch/Pull Time/Commit
-----------------------
2019-12-10_20-00-00
Last Pass
---------
2019-12-10_20-00-00 on (WP_8-12)
Timestamp/Logs
--------------
[2019-12-11 08:58:20,124] 311 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.1:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-list'
[2019-12-11 08:58:21,300] 433 DEBUG MainThread ssh.expect :: Output:
+----+--------------+-------------+----------------+-------------+--------------+
| id | hostname | personality | administrative | operational | availability |
+----+--------------+-------------+----------------+-------------+--------------+
| 1 | controller-0 | controller | unlocked | enabled | available |
| 2 | compute-0 | worker | unlocked | enabled | available |
| 3 | compute-1 | worker | unlocked | enabled | available |
| 4 | controller-1 | controller | unlocked | enabled | available |
+----+--------------+-------------+----------------+-------------+--------------+
[2019-12-11 08:58:22,661] 311 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.1:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-lock compute-0'
[2019-12-11 08:59:40,320] 311 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.1:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-unlock compute-0'
[2019-12-11 09:05:59,264] 311 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.1:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-list'
[2019-12-11 09:06:00,442] 433 DEBUG MainThread ssh.expect :: Output:
+----+--------------+-------------+----------------+-------------+--------------+
| id | hostname | personality | administrative | operational | availability |
+----+--------------+-------------+----------------+-------------+--------------+
| 1 | controller-0 | controller | unlocked | enabled | degraded |
| 2 | compute-0 | worker | unlocked | enabled | available |
| 3 | compute-1 | worker | unlocked | enabled | available |
| 4 | controller-1 | controller | unlocked | enabled | available |
+----+--------------+-------------+----------------+-------------+--------------+
[sysadmin@controller-0 ~(keystone_admin)]$
[2019-12-11 09:11:08,717] 311 DEBUG MainThread ssh.send :: Send 'fm --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.1:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne alarm-list --nowrap --uuid'
[2019-12-11 09:11:09,693] 433 DEBUG MainThread ssh.expect :: Output:
+--------------------------------------+----------+------------------------------------------------------------------------------------------------------------------------+--------------------------------+----------+----------------------------+
| UUID | Alarm ID | Reason Text | Entity ID | Severity | Time Stamp |
+--------------------------------------+----------+------------------------------------------------------------------------------------------------------------------------+--------------------------------+----------+----------------------------+
| 26e10dab-15dd-45ee-b5ac-4ae73bb5db8d | 200.006 | controller-0 is degraded due to the failure of its 'ceph' process. Auto recovery of this major process is in progress. | host=controller-0.process=ceph | major | 2019-12-11T09:00:12.697608 |
+--------------------------------------+----------+------------------------------------------------------------------------------------------------------------------------+--------------------------------+----------+----------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$
Test Activity
-------------
Sanity |
|
| 2019-12-11 18:19:56 |
Ghada Khalil |
starlingx: status |
New |
Triaged |
|
| 2019-12-11 18:20:28 |
Ghada Khalil |
tags |
|
stx.config stx.storage |
|
| 2019-12-11 18:20:29 |
Frank Miller |
bug |
|
|
added subscriber Ovidiu Poncea |
| 2019-12-11 18:20:45 |
Bill Zvonar |
bug |
|
|
added subscriber Bill Zvonar |
| 2019-12-11 18:21:16 |
Ghada Khalil |
starlingx: assignee |
|
Dan Voiculeasa (dvoicule) |
|
| 2019-12-11 18:21:22 |
Ghada Khalil |
starlingx: status |
Triaged |
New |
|
| 2019-12-11 18:21:29 |
Ghada Khalil |
tags |
stx.config stx.storage |
stx.storage |
|
| 2019-12-12 16:34:46 |
Yang Liu |
tags |
stx.storage |
stx.retestneeded stx.storage |
|
| 2019-12-12 20:20:42 |
Bill Zvonar |
removed subscriber Bill Zvonar |
|
|
|
| 2019-12-12 20:20:53 |
Bill Zvonar |
bug |
|
|
added subscriber Daniel Badea |
| 2019-12-16 18:31:00 |
Ghada Khalil |
starlingx: importance |
Undecided |
High |
|
| 2019-12-16 18:31:02 |
Ghada Khalil |
starlingx: status |
New |
Triaged |
|
| 2019-12-16 18:31:08 |
Ghada Khalil |
tags |
stx.retestneeded stx.storage |
stx.3.0 stx.retestneeded stx.storage |
|
| 2019-12-23 15:31:13 |
Peng Peng |
attachment added |
|
ALL_NODES_20191223.094744.tar https://bugs.launchpad.net/starlingx/+bug/1856064/+attachment/5314878/+files/ALL_NODES_20191223.094744.tar |
|
| 2020-02-12 18:46:35 |
Ghada Khalil |
tags |
stx.3.0 stx.retestneeded stx.storage |
stx.3.0 stx.4.0 stx.retestneeded stx.storage |
|
| 2020-02-12 18:47:30 |
Ghada Khalil |
bug |
|
|
added subscriber Bill Zvonar |
| 2020-02-19 16:39:36 |
Dan Voiculeasa |
starlingx: status |
Triaged |
Fix Released |
|
| 2020-02-21 19:11:12 |
Peng Peng |
attachment added |
|
ALL_NODES_20200221.190513.tar https://bugs.launchpad.net/starlingx/+bug/1856064/+attachment/5330117/+files/ALL_NODES_20200221.190513.tar |
|
| 2020-02-26 16:48:53 |
Yang Liu |
starlingx: status |
Fix Released |
Confirmed |
|
| 2020-03-02 18:27:50 |
Ghada Khalil |
removed subscriber Bill Zvonar |
|
|
|
| 2020-03-02 18:28:03 |
Bill Zvonar |
removed subscriber Daniel Badea |
|
|
|
| 2020-03-04 14:20:26 |
Frank Miller |
starlingx: assignee |
Dan Voiculeasa (dvoicule) |
Paul-Ionut Vaduva (pvaduva) |
|
| 2020-03-10 16:06:37 |
OpenStack Infra |
starlingx: status |
Confirmed |
In Progress |
|
| 2020-03-31 14:03:28 |
OpenStack Infra |
starlingx: status |
In Progress |
Fix Released |
|
| 2020-03-31 14:46:22 |
OpenStack Infra |
tags |
stx.3.0 stx.4.0 stx.retestneeded stx.storage |
in-f-centos8 stx.3.0 stx.4.0 stx.retestneeded stx.storage |
|
| 2020-04-21 14:24:52 |
Peng Peng |
tags |
in-f-centos8 stx.3.0 stx.4.0 stx.retestneeded stx.storage |
in-f-centos8 stx.3.0 stx.4.0 stx.storage |
|
| 2020-08-13 20:56:02 |
Bill Zvonar |
tags |
in-f-centos8 stx.3.0 stx.4.0 stx.storage |
in-f-centos8 stx.3.0 stx.4.0 stx.cherrypickneeded stx.storage |
|
| 2020-09-03 18:03:38 |
Bill Zvonar |
tags |
in-f-centos8 stx.3.0 stx.4.0 stx.cherrypickneeded stx.storage |
in-f-centos8 stx.3.0 stx.4.0 stx.storage |
|
