AIO-DX: hypervisor is not up in 5 mins after unlocked standby controller becomes available
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
StarlingX |
Won't Fix
|
Low
|
Frank Miller |
Bug Description
Brief Description
-----------------
After lock/unlock standby controller, standby controller shows down in hypervisor list after 15 mins.
10 mins after host-unlock, host became available. After another 5 mins, hypervisor was still showing down.
Severity
--------
Major
Steps to Reproduce
------------------
TC-name:
Expected Behavior
------------------
Actual Behavior
----------------
Reproducibility
---------------
Reproducible
System Configuration
-------
Two node system
Lab-name: WP_1-2
Branch/Pull Time/Commit
-------
stx master as of 2019-05-18_06-36-50
Last Pass
---------
20190506T233000Z
Timestamp/Logs
--------------
[2019-05-21 13:37:51,407] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://
[2019-05-21 13:38:02,097] 387 DEBUG MainThread ssh.expect :: Output:
+------
| Property | Value |
+------
| action | none |
| administrative | locked |
| availability | online |
[2019-05-21 13:45:18,276] 262 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://
[2019-05-21 13:45:21,240] 387 DEBUG MainThread ssh.expect :: Output:
+----+-
| ID | Hypervisor Hostname | Hypervisor Type | Host IP | State |
+----+-
| 3 | controller-1 | QEMU | 192.168.206.4 | down |
| 7 | controller-0 | QEMU | 192.168.206.3 | up |
+----+-
[2019-05-21 13:50:47,155] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://
[2019-05-21 13:50:49,056] 387 DEBUG MainThread ssh.expect :: Output:
+----+-
| id | hostname | personality | administrative | operational | availability |
+----+-
| 1 | controller-0 | controller | unlocked | enabled | available |
| 2 | controller-1 | controller | unlocked | enabled | available |
+----+-
[2019-05-21 13:51:21,057] 262 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://
[2019-05-21 13:51:23,359] 387 DEBUG MainThread ssh.expect :: Output:
+----+-
| ID | Hypervisor Hostname | Hypervisor Type | Host IP | State |
+----+-
| 3 | controller-1 | QEMU | 192.168.206.4 | down |
| 7 | controller-0 | QEMU | 192.168.206.3 | up |
+----+-
[2019-05-21 13:51:28,469] 262 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://
[2019-05-21 13:51:31,188] 387 DEBUG MainThread ssh.expect :: Output:
+----+-
| ID | Hypervisor Hostname | Hypervisor Type | Host IP | State |
+----+-
| 3 | controller-1 | QEMU | 192.168.206.4 | up |
| 7 | controller-0 | QEMU | 192.168.206.3 | up |
+----+-
Test Activity
-------------
Sanity
tags: | added: stx.sanity |
tags: | added: stx.retestneeded |
tags: | added: stx.containers |
Changed in starlingx: | |
status: | New → Triaged |
importance: | Undecided → Medium |
Changed in starlingx: | |
assignee: | nobody → Bart Wensley (bartwensley) |
summary: |
- Standby controller not up in hypervisor list in 15 mins after host- - unlock + hypervisor is not up in 5 mins after unlocked Standby controller becomes + available |
description: | updated |
summary: |
- hypervisor is not up in 5 mins after unlocked Standby controller becomes - available + AIO-DX: hypervisor is not up in 5 mins after unlocked standby controller + becomes available |
Additional info:
# After unlock, controller-1 became available in system host-show at following time. 192.168. 204.2:5000/ v3 --os-user- domain- name Default --os-project- domain- name Default --os-endpoint-type internalURL --os-region-name RegionOne host-show controller-1' ------- ------- -+----- ------- ------- ------- ------- ------- ------- ------- ------- ------- ---+ ------- ------- -+----- ------- ------- ------- ------- ------- ------- ------- ------- ------- ---+
[2019-05-21 13:45:10,063] 262 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://
+------
| Property | Value |
+------
| action | none |
| administrative | unlocked |
| availability | available |
# It then took another 6 minutes for worker subfunction to be ready and nova hypervisor to be up.
And the complaint is against this extra 6 minutes, which seems to be fairly slow.