overcloud node provision fails on tripleo-quickstart-promote-master-centos9-current-tripleo-delorean-minimal

[Rafael Castillo]

Example logs: https://artifacts.ci.centos.org/rdo/jenkins-tripleo-quickstart-promote-master-centos9-current-tripleo-delorean-minimal-29/undercloud/home/stack/overcloud_node_provision.log

2022-03-01 00:33:28.721428 | 0023b119-6ecf-7c64-dc89-000000000018 | FATAL | Provision instances | localhost | error={"changed": false, "logging": "Created port overcloud-compute-0-ctlplane (UUID ec983189-ecd7-42a8-948b-21b305855133) for node compute-0 (UUID 0b590de5-76c5-40ef-a78e-2fb5acdacada) with {'network_id': '9538f3da-eddc-4920-af60-b7cff411523c', 'name': 'overcloud-compute-0-ctlplane'}\nCreated port overcloud-controller-0-ctlplane (UUID 40f5b039-9d24-4292-a0a3-a546c00d7d57) for node control-0 (UUID 6685330e-f87c-4ea8-b017-2d69042d7e0e) with {'network_id': '9538f3da-eddc-4920-af60-b7cff411523c', 'name': 'overcloud-controller-0-ctlplane'}\nAttached port overcloud-compute-0-ctlplane (UUID ec983189-ecd7-42a8-948b-21b305855133) to node compute-0 (UUID 0b590de5-76c5-40ef-a78e-2fb5acdacada)\nAttached port overcloud-controller-0-ctlplane (UUID 40f5b039-9d24-4292-a0a3-a546c00d7d57) to node control-0 (UUID 6685330e-f87c-4ea8-b017-2d69042d7e0e)\nProvisioning started on node compute-0 (UUID 0b590de5-76c5-40ef-a78e-2fb5acdacada)\nProvisioning started on node control-0 (UUID 6685330e-f87c-4ea8-b017-2d69042d7e0e)\n", "msg": "Node 0b590de5-76c5-40ef-a78e-2fb5acdacada reached failure state \"deploy failed\"; the last error is Failed to prepare to deploy: IPMI call failed: raw 0x00 0x08 0x05 0xa0 0x04 0x00 0x00 0x00."}

Ironic conductor log:
2022-03-01 00:32:31.967 2 WARNING ironic.drivers.modules.ipmitool [req-e8a08813-db2f-4630-b4de-7d77915b1c77 admin - - - -] IPMI Error encountered, retrying "ipmitool -I lanplus -H 192.168.24.1 -L ADMINISTRATOR -p 6230 -U admin -R 1 -N 5 -f /tmp/tmpvnbser0k raw 0x00 0x08 0x05 0xa0 0x04 0x00 0x00 0x00" for node 6685330e-f87c-4ea8-b017-2d69042d7e0e. Error: Unexpected error while running command.
Command: ipmitool -I lanplus -H 192.168.24.1 -L ADMINISTRATOR -p 6230 -U admin -R 1 -N 5 -f /tmp/tmpvnbser0k raw 0x00 0x08 0x05 0xa0 0x04 0x00 0x00 0x00
Exit code: 1
Stdout: ''
Stderr: 'Unable to send RAW command (channel=0x0 netfn=0x0 lun=0x0 cmd=0x8)\n': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2022-03-01 00:32:31.968 2 DEBUG oslo_concurrency.processutils [req-f6fcd18b-72fe-41e1-98ba-6540e15583d7 admin - - - -] CMD "ipmitool -I lanplus -H 192.168.24.1 -L ADMINISTRATOR -p 6231 -U admin -R 1 -N 5 -f /tmp/tmp35xnqa9w raw 0x00 0x08 0x05 0xa0 0x04 0x00 0x00 0x00" returned: 1 in 5.846s execute /usr/lib/python3.9/site-packages/oslo_concurrency/processutils.py:422
2022-03-01 00:32:31.969 2 DEBUG oslo_concurrency.processutils [req-f6fcd18b-72fe-41e1-98ba-6540e15583d7 admin - - - -] 'ipmitool -I lanplus -H 192.168.24.1 -L ADMINISTRATOR -p 6231 -U admin -R 1 -N 5 -f /tmp/tmp35xnqa9w raw 0x00 0x08 0x05 0xa0 0x04 0x00 0x00 0x00' failed. Not Retrying. execute /usr/lib/python3.9/site-packages/oslo_concurrency/processutils.py:473
2022-03-01 00:32:31.969 2 WARNING ironic.drivers.modules.ipmitool [req-f6fcd18b-72fe-41e1-98ba-6540e15583d7 admin - - - -] IPMI Error encountered, retrying "ipmitool -I lanplus -H 192.168.24.1 -L ADMINISTRATOR -p 6231 -U admin -R 1 -N 5 -f /tmp/tmp35xnqa9w raw 0x00 0x08 0x05 0xa0 0x04 0x00 0x00 0x00" for node 0b590de5-76c5-40ef-a78e-2fb5acdacada. Error: Unexpected error while running command.
Command: ipmitool -I lanplus -H 192.168.24.1 -L ADMINISTRATOR -p 6231 -U admin -R 1 -N 5 -f /tmp/tmp35xnqa9w raw 0x00 0x08 0x05 0xa0 0x04 0x00 0x00 0x00
Exit code: 1
Stdout: ''
Stderr: 'Unable to send RAW command (channel=0x0 netfn=0x0 lun=0x0 cmd=0x8)\n': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.

[Ronelle Landy (rlandy)]

Waiting on https://jenkins-cloudsig-ci.apps.ocp.ci.centos.org/job/tripleo-quickstart-promote-master-centos9-current-tripleo-delorean-minimal/31/ to run and see if node provision has cleared

[Ronelle Landy (rlandy)]

https://artifacts.ci.centos.org/rdo/jenkins-tripleo-quickstart-promote-master-centos9-current-tripleo-delorean-minimal-32/undercloud/home/stack/overcloud_node_provision.log

possible new failure:

2022-03-01 21:43:35.938313 | 003d2383-96ad-2b7c-7a70-00000000000c | TASK | Find the growvols utility
[WARNING]: Unhandled error in Python interpreter discovery for host overcloud-
controller-0: Failed to connect to the host via ssh: ssh: connect to host
192.168.24.9 port 22: No route to host

[Rafael Castillo (rafaelcastillo)]

overcloud nodes not coming up: https://review.rdoproject.org/paste/show/253/

[Rafael Castillo (rafaelcastillo)]

New error seems unrelated. Closing this bug for clarity