Rocky periodic job FS001 failed at overcloud prepare image setup giving following errors:
https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky/4820d62/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz#_2018-09-17_02_06_38
2018-09-17 02:06:38 | Exception registering nodes: {u'status': u'FAILED', u'message': [{u'result': u'Node d51d0e4d-c04d-4672-b560-405980bfe993 did not reach state "manageable", the state is "enroll", error: Failed to get power state for node d51d0e4d-c04d-4672-b560-405980bfe993. Error: IPMI call failed: power status.'}, {u'result': u'Node 50ab94b4-9a55-4c1b-9616-26428eec977c did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 50ab94b4-9a55-4c1b-9616-26428eec977c. Error: IPMI call failed: power status.'}, {u'result': u'Node 5a572d9d-590b-4c67-b37e-363ae4234060 did not reach state "manageable", the state is "enroll", error: Failed to get power state for node 5a572d9d-590b-4c67-b37e-363ae4234060. Error: IPMI call failed: power status.'}, {u'result': u'Node d28945f3-06c2-4730-bb9b-d1124a1380f0 did not reach state "manageable", the state is "enroll", error: Failed to get power state for node d28945f3-06c2-4730-bb9b-d1124a1380f0. Error: IPMI call failed: power status.'}], u'result': u'Failure caused by error in tasks: send_message\n\n send_message [task_ex_id=cb21e549-04a7-4a93-9b84-84806d6be2a3] -> Workflow failed due to message status\n [wf_ex_id=31fbb0a4-8e79-4c28-8f16-4c12eeb981d3, idx=0]: Workflow failed due to message status\n'}
While going deep, we found that
https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky/4820d62/logs/undercloud/var/log/extra/docker/containers/ironic_pxe_http/log/ironic/ironic-conductor.log.txt.gz#_2018-09-17_02_06_28_644
2018-09-17 02:06:28.644 7 ERROR ironic.drivers.modules.ipmitool [req-7276def4-b4d4-4695-a060-9b0ccc998e5c 70b4958c371e40e9b34bd1262ff1111e 7a684903197e4a9f8c8c82b1d39c4c53 - default default] IPMI Error while attempting "ipmitool -I lanplus -H 192.168.100.234 -L ADMINISTRATOR -U admin -R 12 -N 5 -f /tmp/tmpz15Zrd power status" for node d28945f3-06c2-4730-bb9b-d1124a1380f0. Error: Unexpected error while running command.
Command: ipmitool -I lanplus -H 192.168.100.234 -L ADMINISTRATOR -U admin -R 12 -N 5 -f /tmp/tmpz15Zrd power status
Exit code: 1
Stdout: u''
Stderr: u'Error: Unable to establish IPMI v2 / RMCP+ session\n': ProcessExecutionError: Unexpected error while running command.
2018-09-17 02:06:28.644 7 WARNING ironic.drivers.modules.ipmitool [req-7276def4-b4d4-4695-a060-9b0ccc998e5c 70b4958c371e40e9b34bd1262ff1111e 7a684903197e4a9f8c8c82b1d39c4c53 - default default] IPMI power status failed for node d28945f3-06c2-4730-bb9b-d1124a1380f0 with error: Unexpected error while running command.
Command: ipmitool -I lanplus -H 192.168.100.234 -L ADMINISTRATOR -U admin -R 12 -N 5 -f /tmp/tmpz15Zrd power status
Exit code: 1
Stdout: u''
Stderr: u'Error: Unable to establish IPMI v2 / RMCP+ session\n'.: ProcessExecutionError: Unexpected error while running command.
2018-09-17 02:06:28.644 7 ERROR ironic.conductor.manager [req-7276def4-b4d4-4695-a060-9b0ccc998e5c 70b4958c371e40e9b34bd1262ff1111e 7a684903197e4a9f8c8c82b1d39c4c53 - default default] Failed to get power state for node d28945f3-06c2-4730-bb9b-d1124a1380f0. Error: IPMI call failed: power status.: IPMIFailure: IPMI call failed: power status.
There might be some issue with Ironic conductor leading to failed to bring node to power status.
We are still seeing the same issue on FS035 https:/ /logs.rdoprojec t.org/openstack -periodic/ git.openstack. org/openstack- infra/tripleo- ci/master/ legacy- periodic- tripleo- ci-centos- 7-ovb-3ctlr_ 1comp-featurese t035-rocky/ ea38bc2/ logs/undercloud /var/log/ containers/ ironic/ ironic- conductor. log.txt. gz#_2018- 09-17_18_ 51_59_180