tripleo

Wallaby: Node provisioning failing, Overcloud nodes not booting properly - System BootOrder not found.

Bug #1975588 reported by Sandeep Yadav on 2022-05-24

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Invalid	Critical	Unassigned	tripleo zed-1

Bug Description

Description:

Wallaby node provisioning failing, Overcloud nodes not booting properly - System BootOrder not found. Initializing defaults.

https://logserver.rdoproject.org/73/26273/23/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/7f810d6/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz

~~~

PLAY [Overcloud Node Grow Volumes] *********************************************
2022-05-24 10:11:23.592002 | fa163ed2-060d-9bc3-e34c-00000000000c | TASK | Wait for provisioned nodes to boot
[WARNING]: Unhandled error in Python interpreter discovery for host overcloud-
controller-2: Failed to connect to the host via ssh: ssh: connect to host
192.168.24.15 port 22: No route to host
[WARNING]: Unhandled error in Python interpreter discovery for host overcloud-
controller-0: Failed to connect to the host via ssh: ssh: connect to host
192.168.24.24 port 22: No route to host
[WARNING]: Unhandled error in Python interpreter discovery for host overcloud-
controller-1: Failed to connect to the host via ssh: ssh: connect to host
192.168.24.10 port 22: No route to host
2022-05-24 10:21:38.230926 | fa163ed2-060d-9bc3-e34c-00000000000c | FATAL | Wait for provisioned nodes to boot | overcloud-controller-1 | error={"changed": false, "elapsed": 614, "msg": "timed out waiting for ping module test: Data could not be sent to remote host
~~~

https://logserver.rdoproject.org/73/26273/23/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/7f810d6/logs/baremetal_23_57893_0-console.log

~~~
[2J[01;01H[=3h[2J[01;01H[0m[35m[40m[0m[37m[40mBdsDxe: loading Boot0001 "UEFI Misc Device" from PciRoot(0x0)/Pci(0x2,0x7)/Pci(0x0,0x0)
BdsDxe: starting Boot0001 "UEFI Misc Device" from PciRoot(0x0)/Pci(0x2,0x7)/Pci(0x0,0x0)
System BootOrder not found. Initializing defaults.
Creating boot entry "Boot0003" with label "CentOS Linux" for file "\EFI\centos\shimx64.efi"

[H[J[1;1H[?25l[m[H[J[1;1H[20;7H[mUse the ^ and v keys to change the selection.
Press 'e' to edit the selected item, or 'c' for a command prompt. [4;80H [7m[4;1H CentOS Stream (5.14.0-96.el9.x86_64) 9 [m[4;79H[m[m[5;1H CentOS Stream (5.14.0-86.el9.x86_64) 9 [m[5;79H[m[m[6;1H CentOS Stream (5.14.0-80.el9.x86_64) 9 [m[6;79H[m[m[7;1H UEFI Firmware Settings [m[7;79H[m[m[8;1H [m[8;79H[m[m[9;1H [m[9;79H[m[m[10;1H [m[10;79H[m[m[11;1H [m[11;79H[m[m[12;1H [m[12;79H[m[m[13;1H [m[13;79H[m[m[14;1H [m[14;79H[m[m[15;1H [m[15;79H[m[m[16;1H [m[16;79H[m[m[17;1H [m[17;79H[m[m[18;1H [m[18;79H[m[18;80H [4;79H[0m[30m[40m[2J[01;01H[0m[37m[40m[21;07HUse the ^ and v keys to change the selection.
Press 'e' to edit the selected item, or 'c' for a command prompt. [04;80H [0m[30m[47m[04;01H CentOS Stream (5.14.0-96.el9.x86_64) 9 [0m[37m[40m[04;79H[05;01H CentOS Stream (5.14.0-86.el9.x86_64) 9 [05;79H[06;01H CentOS Stream (5.14.0-80.el9.x86_64) 9 [06;79H[07;01H UEFI Firmware Settings [07;79H[08;01H [08;79H[09;01H [09;79H[10;01H [10;79H[11;01H [11;79H[12;01H [12;79H[13;01H [13;79H[14;01H [14;79H[15;01H [15;79H[16;01H [16;79H[17;01H [17;79H[18;01H [18;79H[19;01H [19;79H[19;80H [04;79H[23;01H The selected entry will be started automatically in 5s. [04;79H[22;1H The selected entry will be started automatically in 5s. [4;79H[23;01H The selected entry will be started automatically in 4s. [04;79H[22;1H The selected entry will be started automatically in 4s. [4;79H[23;01H The selected entry will be started automatically in 3s. [04;79H[22;1H The selected entry will be started automatically in 3s. [4;79H[23;01H The selected entry will be started automatically in 2s. [04;79H[22;1H The selected entry will be started automatically in 2s. [4;79H[23;01H The selected entry will be started automatically in 1s. [04;79H[22;1H The selected entry will be started automatically in 1s. [4;79H[23;01H The selected entry will be started automatically in 0s. [04;79H[22;1H The selected entry will be started automatically in 0s. [4;79H[0m[30m[40m[2J[01;01H[0m[37m[40m[?25h[H[J[1;1H[H[J[1;1H[0m[30m[40m[2J[01;01H[0m[37m[40m
~~~

Tags:

Revision history for this message

Harald Jensås (harald-jensas) wrote on 2022-05-24:

If I add debug at[1]:

--- a/neutron/services/segments/db.py
+++ b/neutron/services/segments/db.py
@@ -328,18 +328,21 @@ def _add_segment_host_mapping_for_segment(resource, event, trigger,
     context = payload.context
     segment = payload.latest_state
     if not utils.is_session_active(context.session):
+ print('HJENSAS - not utils.is_session_active(context.session)')
         # The session might be in partial rollback state, due to errors in
         # peer callback. In that case, there is no need to add the mapping.
         # Just return here.
         return

Result:

stack@devstack:~$ sudo journalctl -u <email address hidden> -f | grep HJENSAS
May 24 14:47:22 devstack neutron-server[126430]: HJENSAS - not utils.is_session_active(context.session)

https://opendev.org/openstack/neutron/src/branch/master/neutron/services/segments/db.py#L330