Issue was reproduced in IP_5-6 load: 2019-08-19_20-59-00 After 6 mins, hypervisor is up.
[2019-08-20 09:29:35,346] 301 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne hypervisor list' [2019-08-20 09:29:38,493] 423 DEBUG MainThread ssh.expect :: Output: +----+---------------------+-----------------+---------------+-------+ | ID | Hypervisor Hostname | Hypervisor Type | Host IP | State | +----+---------------------+-----------------+---------------+-------+ | 4 | controller-0 | QEMU | 192.168.205.3 | up | | 6 | controller-1 | QEMU | 192.168.205.4 | up | +----+---------------------+-----------------+---------------+-------+
[2019-08-20 09:32:29,503] 301 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-lock controller-0'
[2019-08-20 09:33:40,287] 301 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-unlock controller-0'
[2019-08-20 09:42:30,969] 301 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne hypervisor list' [2019-08-20 09:42:34,893] 423 DEBUG MainThread ssh.expect :: Output: +----+---------------------+-----------------+---------------+-------+ | ID | Hypervisor Hostname | Hypervisor Type | Host IP | State | +----+---------------------+-----------------+---------------+-------+ | 4 | controller-0 | QEMU | 192.168.205.3 | down | | 6 | controller-1 | QEMU | 192.168.205.4 | up | +----+---------------------+-----------------+---------------+-------+
[2019-08-20 09:47:28,606] 301 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne hypervisor list' [2019-08-20 09:47:31,561] 423 DEBUG MainThread ssh.expect :: Output: +----+---------------------+-----------------+---------------+-------+ | ID | Hypervisor Hostname | Hypervisor Type | Host IP | State | +----+---------------------+-----------------+---------------+-------+ | 4 | controller-0 | QEMU | 192.168.205.3 | down | | 6 | controller-1 | QEMU | 192.168.205.4 | up | +----+---------------------+-----------------+---------------+-------+
[2019-08-20 09:47:57,474] 301 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne hypervisor list' [2019-08-20 09:48:00,972] 423 DEBUG MainThread ssh.expect :: Output: +----+---------------------+-----------------+---------------+-------+ | ID | Hypervisor Hostname | Hypervisor Type | Host IP | State | +----+---------------------+-----------------+---------------+-------+ | 4 | controller-0 | QEMU | 192.168.205.3 | up | | 6 | controller-1 | QEMU | 192.168.205.4 | up | +----+---------------------+-----------------+---------------+-------+
Issue was reproduced in IP_5-6 load: 2019-08-19_20-59-00
After 6 mins, hypervisor is up.
[2019-08-20 09:29:35,346] 301 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http:// keystone. openstack. svc.cluster. local/v3 --os-user- domain- name Default --os-project- domain- name Default --os-identity- api-version 3 --os-interface internal --os-region-name RegionOne hypervisor list' ------- ------- ------+ ------- ------- ---+--- ------- -----+- ------+ ------- ------- ------+ ------- ------- ---+--- ------- -----+- ------+ ------- ------- ------+ ------- ------- ---+--- ------- -----+- ------+
[2019-08-20 09:29:38,493] 423 DEBUG MainThread ssh.expect :: Output:
+----+-
| ID | Hypervisor Hostname | Hypervisor Type | Host IP | State |
+----+-
| 4 | controller-0 | QEMU | 192.168.205.3 | up |
| 6 | controller-1 | QEMU | 192.168.205.4 | up |
+----+-
[2019-08-20 09:32:29,503] 301 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http:// 192.168. 204.2:5000/ v3 --os-user- domain- name Default --os-project- domain- name Default --os-endpoint-type internalURL --os-region-name RegionOne host-lock controller-0'
[2019-08-20 09:33:40,287] 301 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http:// 192.168. 204.2:5000/ v3 --os-user- domain- name Default --os-project- domain- name Default --os-endpoint-type internalURL --os-region-name RegionOne host-unlock controller-0'
[2019-08-20 09:42:30,969] 301 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http:// keystone. openstack. svc.cluster. local/v3 --os-user- domain- name Default --os-project- domain- name Default --os-identity- api-version 3 --os-interface internal --os-region-name RegionOne hypervisor list' ------- ------- ------+ ------- ------- ---+--- ------- -----+- ------+ ------- ------- ------+ ------- ------- ---+--- ------- -----+- ------+ ------- ------- ------+ ------- ------- ---+--- ------- -----+- ------+
[2019-08-20 09:42:34,893] 423 DEBUG MainThread ssh.expect :: Output:
+----+-
| ID | Hypervisor Hostname | Hypervisor Type | Host IP | State |
+----+-
| 4 | controller-0 | QEMU | 192.168.205.3 | down |
| 6 | controller-1 | QEMU | 192.168.205.4 | up |
+----+-
[2019-08-20 09:47:28,606] 301 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http:// keystone. openstack. svc.cluster. local/v3 --os-user- domain- name Default --os-project- domain- name Default --os-identity- api-version 3 --os-interface internal --os-region-name RegionOne hypervisor list' ------- ------- ------+ ------- ------- ---+--- ------- -----+- ------+ ------- ------- ------+ ------- ------- ---+--- ------- -----+- ------+ ------- ------- ------+ ------- ------- ---+--- ------- -----+- ------+
[2019-08-20 09:47:31,561] 423 DEBUG MainThread ssh.expect :: Output:
+----+-
| ID | Hypervisor Hostname | Hypervisor Type | Host IP | State |
+----+-
| 4 | controller-0 | QEMU | 192.168.205.3 | down |
| 6 | controller-1 | QEMU | 192.168.205.4 | up |
+----+-
[2019-08-20 09:47:57,474] 301 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http:// keystone. openstack. svc.cluster. local/v3 --os-user- domain- name Default --os-project- domain- name Default --os-identity- api-version 3 --os-interface internal --os-region-name RegionOne hypervisor list' ------- ------- ------+ ------- ------- ---+--- ------- -----+- ------+ ------- ------- ------+ ------- ------- ---+--- ------- -----+- ------+ ------- ------- ------+ ------- ------- ---+--- ------- -----+- ------+
[2019-08-20 09:48:00,972] 423 DEBUG MainThread ssh.expect :: Output:
+----+-
| ID | Hypervisor Hostname | Hypervisor Type | Host IP | State |
+----+-
| 4 | controller-0 | QEMU | 192.168.205.3 | up |
| 6 | controller-1 | QEMU | 192.168.205.4 | up |
+----+-