diskfilter writes are not supported. - Timeout waiting for provisioned nodes to become available in fs1 wallaby

[Ananya Banerjee]

tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby is failing in TASK [tripleo.operator.tripleo_overcloud_node_provision : overcloud node provision] with

 FATAL | Wait for provisioned nodes to boot | overcloud-controller-1 | error={"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python3"}, "changed": false, "elapsed": 601, "msg": "Timeout waiting for provisioned nodes to become available"}

Logs:
https://logserver.rdoproject.org/84/820584/16/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/f114382/job-output.txt
https://logserver.rdoproject.org/49/821849/8/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/a4ecead/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz

[Ananya Banerjee (frenzyfriday)]

Baremetal console logs:
https://logserver.rdoproject.org/49/821849/8/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/a4ecead/logs/baremetal_8_68959_0-console.log

[H[J[1;1Herror: ../../grub-core/disk/diskfilter.c:916:diskfilter writes are not
supported.
[?25l[m[H[J[1;1H[20;7H[mUse the ^ and v keys to change the selection.
      Press 'e' to edit the selected item, or 'c' for a command prompt. [4;80H [7m[4;1H CentOS Stream (5.14.0-62.el9.x86_64) 9 [m[4;79H[m[m[5;1H CentOS Stream (5.14.0-58.el9.x86_64) 9 [m[5;79H[m[m[6;1H [m[6;79H[m[m[7;1H [m[7;79H[m[m[8;1H [m[8;79H[m[m[9;1H [m[9;79H[m[m[10;1H [m[10;79H[m[m[11;1H [m[11;79H[m[m[12;1H [m[12;79H[m[m[13;1H [m[13;79H[m[m[14;1H [m[14;79H[m[m[15;1H [m[15;79H[m[m[16;1H [m[16;79H[m[m[17;1H [m[17;79H[m[m[18;1H [m[18;79H[m[18;80H [4;79H[22;1H The selected entry will be started automatically in 5s. [4;79H[22;1H The selected entry will be started automatically in 4s. [4;79H[22;1H The selected entry will be started automatically in 3s. [4;79H[22;1H The selected entry will be started automatically in 2s. [4;79H[22;1H The selected entry will be started automatically in 1s. [4;79H[22;1H The selected entry will be started automatically in 0s. [4;79H[?25h[H[J[1;1H[H[J[1;1H

[Sandeep Yadav (sandeepyadav93)]

Again seeing this is check job for master and wallaby:-

wallaby:-

https://logserver.rdoproject.org/64/841064/2/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/2d13c66/logs/baremetal_2_57662_0-console.log

master:-

https://logserver.rdoproject.org/64/841064/2/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001/d84b9cf/logs/baremetal_2_22143_0-console.log

Pinged Ananya to understand previous resolution as it was not clear from this bug comment history:-

~~~~
ysandeep|rover> frenzy_friday, hey o/ what was the resolution of this bug: https://bugs.launchpad.net/tripleo/+bug/1961811 ? Did it resolved on itself?
<ysandeep|rover> I am again seeing this today: https://logserver.rdoproject.org/64/841064/2/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001/d84b9cf/logs/baremetal_2_22143_3-console.log
<frenzy_friday> ysandeep|rover, checking
<frenzy_friday> ysandeep|rover, https://trello.com/c/TpHRXYv5/2366-cixlp1961811tripleociproa-diskfilter-writes-are-not-supported-timeout-waiting-for-provisioned-nodes-to-become-available-in-fs1-w this was the activity on the bug
<frenzy_friday> https://review.opendev.org/c/openstack/diskimage-builder/+/830900, https://review.opendev.org/q/topic:bootloader-nopart
<ysandeep|rover> frenzy_friday: thanks! checking
~~~

[Dariusz Smigiel (smigiel-dariusz)]

Change to enable ipxe uefi boot on check jobs: https://review.rdoproject.org/r/c/rdo-jobs/+/42673 got merged.

It should unblock the gate.

[Sandeep Yadav (sandeepyadav93)]

We are not seeing similiar issue in check after merge of https://review.rdoproject.org/r/c/rdo-jobs/+/42673

[Rabi Mishra (rabi)]

Noticed again https://logserver.rdoproject.org/60/844960/4/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001/dded39b/logs/baremetal_4_10007_0-console.log

[Rabi Mishra (rabi)]

Still happening https://logserver.rdoproject.org/78/845878/2/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001/92bd882/logs/baremetal_2_34705_0-console.log