StarlingX

Bug #1829931
Comment #6

Comment 6 for bug 1829931

Revision history for this message

Peng Peng (ppeng) wrote on 2019-05-24: Re: Standby controller not up in hypervisor list in 15 mins after host-unlock

Yes, it is also happened on standard lab WCP_99-103
Lab: WCP_99_103
Load: 2019-05-21_14-14-17

[2019-05-22 08:41:47,154] 262 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne hypervisor list'
[2019-05-22 08:41:48,764] 387 DEBUG MainThread ssh.expect :: Output:
+----+---------------------+-----------------+-----------------+-------+
| ID | Hypervisor Hostname | Hypervisor Type | Host IP | State |
+----+---------------------+-----------------+-----------------+-------+
| 4 | compute-2 | QEMU | 192.168.206.84 | up |
| 7 | compute-1 | QEMU | 192.168.206.43 | up |
| 10 | compute-0 | QEMU | 192.168.206.123 | up |
+----+---------------------+-----------------+-----------------+-------+

[2019-05-22 08:49:27,127] 139 INFO MainThread host_helper.reboot_hosts:: Rebooting active controller: controller-0
[2019-05-22 08:49:27,127] 262 DEBUG MainThread ssh.send :: Send 'sudo reboot -f'

[2019-05-22 09:10:29,994] 262 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne hypervisor list'
[2019-05-22 09:12:52,140] 387 DEBUG MainThread ssh.expect :: Output:
Unable to establish connection to http://nova-api-proxy.openstack.svc.cluster.local:8774/v2.1/dffd81c66f134c40bed770a1ff06a845/os-hypervisors/detail: HTTPConnectionPool(host='nova-api-proxy.openstack.svc.cluster.local', port=8774): Max retries exceeded with url: /v2.1/dffd81c66f134c40bed770a1ff06a845/os-hypervisors/detail (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f4e0deefdd0>: Failed to establish a new connection: [Errno 110] Connection timed out',))
controller-1:~$

Yes, it is also happened on standard lab WCP_99-103
Lab: WCP_99_103
Load: 2019-05-21_14-14-17

[2019-05-22 08:41:47,154] 262  DEBUG MainThread ssh.send    :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne hypervisor list'
[2019-05-22 08:41:48,764] 387  DEBUG MainThread ssh.expect  :: Output: 
+----+---------------------+-----------------+-----------------+-------+
| ID | Hypervisor Hostname | Hypervisor Type | Host IP         | State |
+----+---------------------+-----------------+-----------------+-------+
|  4 | compute-2           | QEMU            | 192.168.206.84  | up    |
|  7 | compute-1           | QEMU            | 192.168.206.43  | up    |
| 10 | compute-0           | QEMU            | 192.168.206.123 | up    |
+----+---------------------+-----------------+-----------------+-------+

[2019-05-22 08:49:27,127] 139  INFO  MainThread host_helper.reboot_hosts:: Rebooting active controller: controller-0
[2019-05-22 08:49:27,127] 262  DEBUG MainThread ssh.send    :: Send 'sudo reboot -f'

[2019-05-22 09:10:29,994] 262  DEBUG MainThread ssh.send    :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne hypervisor list'
[2019-05-22 09:12:52,140] 387  DEBUG MainThread ssh.expect  :: Output: 
Unable to establish connection to http://nova-api-proxy.openstack.svc.cluster.local:8774/v2.1/dffd81c66f134c40bed770a1ff06a845/os-hypervisors/detail: HTTPConnectionPool(host='nova-api-proxy.openstack.svc.cluster.local', port=8774): Max retries exceeded with url: /v2.1/dffd81c66f134c40bed770a1ff06a845/os-hypervisors/detail (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f4e0deefdd0>: Failed to establish a new connection: [Errno 110] Connection timed out',))
controller-1:~$