centos 9 master/wallaby OVB check jobs are failing node provision - no route to host

[Ronelle Landy]

centos 9 master/wallaby OVB check jobs are failing the node provision step (in Overcloud Node Grow Volumes) with a 'no route to host' error:

PLAY [Overcloud Node Grow Volumes] *********************************************
2022-02-10 10:01:37.969659 | fa163e53-3c0b-2b3b-2b30-00000000000c | TASK | Wait for provisioned nodes to boot
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: OSError: [Errno 113] No route to host
2022-02-10 10:11:14.654638 | fa163e53-3c0b-2b3b-2b30-00000000000c | FATAL | Wait for provisioned nodes to boot | overcloud-controller-0 | error={"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python3"}, "changed": false, "module_stderr": "Traceback (most recent call last):\n File \"<stdin>\", line 107, in <module>\n File \"<stdin>\", line 99, in _ansiballz_main\n File \"<stdin>\", line 47, in invoke_module\n File \"/usr/lib64/python3.9/runpy.py\", line 210, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib64/python3.9/runpy.py\", line 97, in _run_module_code\n _run_code(code, mod_globals, init_globals,\n File \"/usr/lib64/python3.9/runpy.py\", line 87, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_wait_for_payload_k1o8x7ny/ansible_wait_for_payload.zip/ansible/modules/wait_for.py\", line 682, in <module>\n File \"/tmp/ansible_wait_for_payload_k1o8x7ny/ansible_wait_for_payload.zip/ansible/modules/wait_for.py\", line 612, in main\nOSError: [Errno 113] No route to host\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

However on master:

PLAY [Overcloud Node Grow Volumes] *********************************************
2022-02-10 10:43:22.455318 | fa163e1a-46ca-7598-5c3f-00000000000c | TASK | Wait for provisioned nodes to boot
2022-02-10 10:53:24.590799 | fa163e1a-46ca-7598-5c3f-00000000000c | FATAL | Wait for provisioned nodes to boot | overcloud-controller-0 | error={"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python3"}, "changed": false, "elapsed": 601, "msg": "Timeout waiting for provisioned nodes to become available"}

Example logs:

https://logserver.rdoproject.org/71/828671/1/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/eda3778/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz

https://logserver.rdoproject.org/openstack-regular/opendev.org/openstack/tripleo-ci/master/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001/51a3274/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz

Job history:

https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby

The failures looks to have started on 02/09

[Harald Jensås (harald-jensas)]

https://logserver.rdoproject.org/71/828671/1/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/eda3778/logs/baremetal_1_25010_0-console.log
https://logserver.rdoproject.org/71/828671/1/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/eda3778/logs/baremetal_1_25010_1-console.log
https://logserver.rdoproject.org/71/828671/1/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/eda3778/logs/baremetal_1_25010_2-console.log
https://logserver.rdoproject.org/71/828671/1/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/eda3778/logs/baremetal_1_25010_3-console.log

Baremetal nodes does not boot grup issue.

[H[J[1;1Herror: ../../grub-core/disk/diskfilter.c:916:diskfilter writes are not
supported.
[?25l[m[H[J[1;1H[20;7H[mUse the ^ and v keys to change the selection.
      Press 'e' to edit the selected item, or 'c' for a command prompt. [4;80H [m[4;1H CentOS Stream (5.14.0-52.el9.x86_64) 9 [m[4;79H[m[7m[5;1H CentOS Stream (5.14.0-55.el9.x86_64) 9 [m[5;79H[m[m[6;1H [m[6;79H[m[m[7;1H [m[7;79H[m[m[8;1H [m[8;79H[m[m[9;1H [m[9;79H[m[m[10;1H [m[10;79H[m[m[11;1H [m[11;79H[m[m[12;1H [m[12;79H[m[m[13;1H [m[13;79H[m[m[14;1H [m[14;79H[m[m[15;1H [m[15;79H[m[m[16;1H [m[16;79H[m[m[17;1H [m[17;79H[m[m[18;1H [m[18;79H[m[18;80H [5;79H[22;1H The selected entry will be started automatically in 5s. [5;79H[22;1H The selected entry will be started automatically in 4s. [5;79H[22;1H The selected entry will be started automatically in 3s. [5;79H[22;1H The selected entry will be started automatically in 2s. [5;79H[22;1H The selected entry will be started automatically in 1s. [5;79H[22;1H The selected entry will be started automatically in 0s. [5;79H[?25h[H[J[1;1H[H[J[1;1H

[Harald Jensås (harald-jensas)]

Do we need to backport https://review.opendev.org/c/openstack/openstack-virtual-baremetal/+/810332
to stable/2.0 openstack-virtual-baremetal?

[Harald Jensås (harald-jensas)]

I was not able to reproduce this issue on OVB locally.

I used repos:
  tripleo-repos current-tripleo ceph
Images:
  https://images.rdoproject.org/centos9/master/rdo_trunk/current-tripleo/ironic-python-agent.tar
  https://images.rdoproject.org/centos9/master/rdo_trunk/current-tripleo/overcloud-hardened-uefi-full.qcow2

[chandan kumar (chkumar246)]

Here is the proposed fix: https://review.opendev.org/c/openstack/diskimage-builder/+/828617 and testing it via testproject here: https://review.rdoproject.org/r/c/testproject/+/39461

[chandan kumar (chkumar246)]

Based on testproject:
https://review.rdoproject.org/r/c/testproject/+/39461

tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby SUCCESS 2h 40m 43s
tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001 SUCCESS 2h 38m 55s