200.006 alarm "controller-1 is degraded due to the failure of its 'ceph' process"
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
StarlingX |
Fix Released
|
Medium
|
Elena Taivan |
Bug Description
Brief Description
-----------------
On a regular system, after lock/unlock one node (compute or controller), alarm 200.006 | "controller-0 is degraded due to the failure of its 'ceph' process. Auto recovery of this major process is in progress." is raised.
Severity
--------
Major
Steps to Reproduce
------------------
lock compute node
unlock compute node
check alarm-list
TC-name: mtc/test_
Expected Behavior
------------------
no 200.006 alarm
Actual Behavior
----------------
200.006 alarm raised
Reproducibility
---------------
Intermittent 2/3 in sanity run
System Configuration
-------
Multi-node system
Lab-name: WCP_71-75
Branch/Pull Time/Commit
-------
master as of 2020-01-20_00-10-00
Last Pass
---------
2020-01-16_22-02-50
Timestamp/Logs
--------------
[2020-01-20 15:53:49,593] 314 DEBUG MainThread ssh.send :: Send 'fm --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://[face::1]:5000/v3 --os-user-
[2020-01-20 15:53:50,644] 436 DEBUG MainThread ssh.expect :: Output:
[sysadmin@
[2020-01-20 15:54:38,599] 314 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://[face::1]:5000/v3 --os-user-
[2020-01-20 15:54:39,738] 436 DEBUG MainThread ssh.expect :: Output:
+----+-
| id | hostname | personality | administrative | operational | availability |
+----+-
| 1 | controller-0 | controller | unlocked | enabled | available |
| 2 | compute-0 | worker | unlocked | enabled | available |
| 3 | compute-1 | worker | unlocked | enabled | available |
| 4 | compute-2 | worker | unlocked | enabled | available |
| 5 | controller-1 | controller | unlocked | enabled | available |
+----+-
[2020-01-20 15:54:41,112] 314 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://[face::1]:5000/v3 --os-user-
[2020-01-20 15:56:38,659] 314 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://[face::1]:5000/v3 --os-user-
[2020-01-20 16:02:15,135] 314 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://[face::1]:5000/v3 --os-user-
[2020-01-20 16:02:16,317] 436 DEBUG MainThread ssh.expect :: Output:
+----+-
| id | hostname | personality | administrative | operational | availability |
+----+-
| 1 | controller-0 | controller | unlocked | enabled | degraded |
| 2 | compute-0 | worker | unlocked | enabled | available |
| 3 | compute-1 | worker | unlocked | enabled | available |
| 4 | compute-2 | worker | unlocked | enabled | available |
| 5 | controller-1 | controller | unlocked | enabled | available |
+----+-
[2020-01-20 16:02:19,034] 314 DEBUG MainThread ssh.send :: Send 'fm --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://[face::1]:5000/v3 --os-user-
[2020-01-20 16:02:20,042] 436 DEBUG MainThread ssh.expect :: Output:
+------
| UUID | Alarm ID | Reason Text | Entity ID | Severity | Time Stamp |
+------
| 13ff9738-
| 2a258bf7-
| 05853c2b-
| e308f8d2-
| 0a9c655b-
| ac178ecc-
+------
[sysadmin@
Test Activity
-------------
Sanity
Changed in starlingx: | |
assignee: | nobody → Elena Taivan (etaivan) |
Marking as stx.4.0 / medium priority - intermittent issue related to ceph; needs further investigation