Issue was reproduced on Lab: WCP_76_77 Load: 2019-09-09_00-10-00 Job: Titanium_R6_build
[2019-09-10 08:22:07,526] 301 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-swact controller-1'
[sysadmin@controller-1 ~(keystone_admin)]$ [2019-09-10 08:22:31,455] 301 DEBUG MainThread ssh.send :: Send '' [2019-09-10 08:22:34,559] 275 INFO MainThread ssh.wait_for_disconnect:: ssh session to 128.224.150.5 disconnected [2019-09-10 08:22:34,560] 1572 INFO MainThread host_helper.wait_for_swact_complete:: ssh to 128.224.150.5 OAM floating IP disconnected, indicating swact initiated. [2019-09-10 08:23:04,590] 151 INFO MainThread ssh.connect :: Attempt to connect to host - 128.224.150.5 [2019-09-10 08:23:06,099] 301 DEBUG MainThread ssh.send :: Send '' [2019-09-10 08:23:06,204] 423 DEBUG MainThread ssh.expect :: Output: controller-0:~$
[2019-09-10 08:25:51,734] 301 DEBUG MainThread ssh.send :: Send 'fm --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne alarm-list --nowrap --uuid' [2019-09-10 08:25:53,012] 423 DEBUG MainThread ssh.expect :: Output: +--------------------------------------+----------+---------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------+----------+----------------------------+ | UUID | Alarm ID | Reason Text | Entity ID | Severity | Time Stamp | +--------------------------------------+----------+---------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------+----------+----------------------------+ | 32d95c7d-f2d7-4c51-917c-9a122b6a6923 | 400.001 | Service group controller-services failure; ceph-mon(disabling, failed) | service_domain=controller.service_group=controller-services.host=controller-0 | critical | 2019-09-10T08:25:51.606031 | | 2177bd57-2a1c-43fd-b54f-41931c6221bc | 400.002 | Service group controller-services has no active members available; expected 1 active member | service_domain=controller.service_group=controller-services | critical | 2019-09-10T08:22:30.015971 | | 4c643552-1006-48a2-8234-d6d6d3f3fbf7 | 100.114 | NTP address 2607:5300:201:3 is not a valid or a reachable NTP server. | host=controller-1.ntp=2607:5300:201:3 | minor | 2019-09-10T08:08:12.089195 | +--------------------------------------+----------+---------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------+----------+----------------------------+ controller-0:~$
[2019-09-10 08:25:58,121] 301 DEBUG MainThread ssh.send :: Send 'fm --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne alarm-list --nowrap --uuid' [2019-09-10 08:27:15,408] 387 WARNING MainThread ssh.expect :: EOF caught.
controller-1:~$ [2019-09-10 08:27:21,262] 183 INFO MainThread ssh.connect :: Login successful!
Issue was reproduced on
Lab: WCP_76_77
Load: 2019-09-09_00-10-00
Job: Titanium_R6_build
[2019-09-10 08:22:07,526] 301 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http:// 192.168. 204.2:5000/ v3 --os-user- domain- name Default --os-project- domain- name Default --os-endpoint-type internalURL --os-region-name RegionOne host-swact controller-1'
[sysadmin@ controller- 1 ~(keystone_admin)]$ for_disconnect: : ssh session to 128.224.150.5 disconnected wait_for_ swact_complete: : ssh to 128.224.150.5 OAM floating IP disconnected, indicating swact initiated.
[2019-09-10 08:22:31,455] 301 DEBUG MainThread ssh.send :: Send ''
[2019-09-10 08:22:34,559] 275 INFO MainThread ssh.wait_
[2019-09-10 08:22:34,560] 1572 INFO MainThread host_helper.
[2019-09-10 08:23:04,590] 151 INFO MainThread ssh.connect :: Attempt to connect to host - 128.224.150.5
[2019-09-10 08:23:06,099] 301 DEBUG MainThread ssh.send :: Send ''
[2019-09-10 08:23:06,204] 423 DEBUG MainThread ssh.expect :: Output:
controller-0:~$
[2019-09-10 08:25:51,734] 301 DEBUG MainThread ssh.send :: Send 'fm --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http:// 192.168. 204.2:5000/ v3 --os-user- domain- name Default --os-project- domain- name Default --os-endpoint-type internalURL --os-region-name RegionOne alarm-list --nowrap --uuid' ------- ------- ------- ------- ----+-- ------- -+----- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ----+-- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- +------ ----+-- ------- ------- ------- -----+ ------- ------- ------- ------- ----+-- ------- -+----- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ----+-- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- +------ ----+-- ------- ------- ------- -----+ f2d7-4c51- 917c-9a122b6a69 23 | 400.001 | Service group controller-services failure; ceph-mon(disabling, failed) | service_ domain= controller. service_ group=controlle r-services. host=controller -0 | critical | 2019-09- 10T08:25: 51.606031 | 2a1c-43fd- b54f-41931c6221 bc | 400.002 | Service group controller-services has no active members available; expected 1 active member | service_ domain= controller. service_ group=controlle r-services | critical | 2019-09- 10T08:22: 30.015971 | 1006-48a2- 8234-d6d6d3f3fb f7 | 100.114 | NTP address 2607:5300:201:3 is not a valid or a reachable NTP server. | host=controller -1.ntp= 2607:5300: 201:3 | minor | 2019-09- 10T08:08: 12.089195 | ------- ------- ------- ------- ----+-- ------- -+----- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ----+-- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- +------ ----+-- ------- ------- ------- -----+
[2019-09-10 08:25:53,012] 423 DEBUG MainThread ssh.expect :: Output:
+------
| UUID | Alarm ID | Reason Text | Entity ID | Severity | Time Stamp |
+------
| 32d95c7d-
| 2177bd57-
| 4c643552-
+------
controller-0:~$
[2019-09-10 08:25:58,121] 301 DEBUG MainThread ssh.send :: Send 'fm --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http:// 192.168. 204.2:5000/ v3 --os-user- domain- name Default --os-project- domain- name Default --os-endpoint-type internalURL --os-region-name RegionOne alarm-list --nowrap --uuid'
[2019-09-10 08:27:15,408] 387 WARNING MainThread ssh.expect :: EOF caught.
controller-1:~$
[2019-09-10 08:27:21,262] 183 INFO MainThread ssh.connect :: Login successful!