Comment 2 for bug 2034704

>> Openstack-virtual-baremetal last saw changes back in 2022, so could an underlying libvirt change have broken things?

We have been experiencing this intermittent problem with jobs running in both the Vexxhost and PSI clouds. The first failure we noticed occurred on August 28th in the Vexxhost cloud and on August 29th in the PSI internal cloud. Given that the issue started around the same time and the possibility of both PSI and Vexxhost upgrading their infrastructure is low, I wonder if the issue is related to a different layer.

Vexx first failure, 28th

https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-wallaby/d9150b2/job-output.txt
~~~
2023-08-28 18:52:33.799837 | primary | TASK [tripleo.operator.tripleo_overcloud_node_provision : overcloud node provision] ***
2023-08-28 18:52:33.799887 | primary | Monday 28 August 2023 18:52:33 -0400 (0:00:01.751) 0:50:15.892 *********
2023-08-28 18:52:47.039195 | primary | ASYNC POLL on undercloud: jid=527423317646.197406 started=1 finished=0
.
.
2023-08-28 18:59:23.926089 | primary | fatal: [undercloud]: FAILED! => {"ansible_job_id": "527423317646.197406", "changed": false, "cmd": "source /home/zuul/stackrc; openstack overcloud node provision -o $PROVISION_OUTPUT --stack $PROVISION_STACK /home/zuul/overcloud_baremetal_deploy.yaml >/home/zuul/overcloud_node_provision.log 2>&1", "delta": "0:06:36.565803", "end": "2023-08-28 22:59:12.256865", "finished": 1, "msg": "non-zero return code", "rc": 1, "start": "2023-08-28 22:52:35.691062", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
~~~

PSI first failure, 29th

https://sf.hosted.upshift.rdu2.redhat.com/logs/44/49544/24/check-rdo/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-internal-wallaby/5c13a92/job-output.txt

2023-08-29 19:00:41.515201 | primary | fatal: [undercloud]: FAILED! => {"ansible_job_id": "966085717323.182995", "changed": false, "cmd": "source /home/zuul/stackrc; openstack overcloud node provision -o $PROVISION_OUTPUT --stack $PROVISION_STACK /home/zuul/overcloud_baremetal_deploy.yaml >/home/zuul/overcloud_node_provision.log 2>&1", "delta": "0:08:23.094709", "end": "2023-08-29 23:00:39.179603", "finished": 1, "msg": "non-zero return code", "rc": 1, "start": "2023-08-29 22:52:16.084894", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

Sharing build history for fs001:-

https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-internal-wallaby
https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby