tripleo-quickstart fails on "Get undercloud vm ip address"

[Juan Badia Payno]

Deploying a lab with tripleo-quickstart (master-tripleo-ci) failed on get undercloud vm ip address:

./quickstart.sh -X -n -R master-tripleo-ci --tags all -T none --nodes config/nodes/3ctlr_2comp_3ceph.yml -p quickstart.yml 127.0.0.2
..
..
..
TASK [setup/undercloud : Get undercloud vm ip address] ***************************************************************************************************************************************************************************************
task path: /root/tripleo-quickstart/roles/libvirt/setup/undercloud/tasks/main.yml:376
Tuesday 15 June 2021 08:22:53 -0400 (0:00:00.947) 0:11:58.829 **********
FAILED - RETRYING: Get undercloud vm ip address (20 retries left).
...
...
FAILED - RETRYING: Get undercloud vm ip address (1 retries left).
fatal: [127.0.0.2]: FAILED! => {
    "attempts": 20,
    "changed": true,
    "rc": 1
}
STDOUT:
undercloud ip is not available
STDERR:
Connection to 127.0.0.2 closed.
MSG:
non-zero return code

PLAY RECAP ***********************************************************************************************************************************************************************************************************************************
127.0.0.1 : ok=16 changed=5 unreachable=0 failed=0 skipped=23 rescued=0 ignored=0
127.0.0.2 : ok=124 changed=55 unreachable=0 failed=1 skipped=116 rescued=0 ignored=4

[root@undercloud ~]# nmcli con

[root@undercloud ~]# nmcli dev
DEVICE TYPE STATE CONNECTION
eth0 ethernet disconnected --
eth1 ethernet disconnected --
lo loopback unmanaged --

From /var/lib/messages
Jun 14 18:46:51 undercloud sssd[nss][1084]: Starting up
Jun 14 18:46:51 undercloud ovs-ctl[1028]: /etc/openvswitch/conf.db does not exist ... (warning).
Jun 14 18:46:51 undercloud systemd[1]: Started System Security Services Daemon.
Jun 14 18:46:51 undercloud systemd[1]: Reached target User and Group Name Lookups.
Jun 14 18:46:51 undercloud systemd[1]: Starting Login Service...
Jun 14 18:46:51 undercloud ovs-ctl[1028]: Creating empty database /etc/openvswitch/conf.db [ OK ]
Jun 14 18:46:51 undercloud systemd-logind[1098]: New seat seat0.
Jun 14 18:46:51 undercloud systemd-logind[1098]: Watching system buttons on /dev/input/event0 (Power Button)
Jun 14 18:46:51 undercloud systemd-logind[1098]: Watching system buttons on /dev/input/event1 (AT Translated Set 2 keyboard)
Jun 14 18:46:51 undercloud systemd[1]: Started Login Service.
Jun 14 18:46:51 undercloud ovs-ctl[1028]: Starting ovsdb-server [ OK ]
Jun 14 18:46:51 undercloud ovs-vsctl[1105]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait -- init -- set Open_vSwitch . db-version=8.2.0
Jun 14 18:46:51 undercloud ovs-vsctl[1116]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait set Open_vSwitch . ovs-version=2.13.4 "external-ids:system-id=\"f7ef0144-e554-455f-9d89-78e3edcf605e\"" "external-ids:rundir=\"/var/run/openvswitch\"" "system-type=\"centos\"" "system-version=\"8\""
Jun 14 18:46:51 undercloud ovs-ctl[1028]: Configuring Open vSwitch system IDs [ OK ]
Jun 14 18:46:51 undercloud ovs-vsctl[1122]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait add Open_vSwitch . external-ids hostname=undercloud
Jun 14 18:46:51 undercloud ovs-ctl[1028]: Enabling remote OVSDB managers [ OK ]
Jun 14 18:46:51 undercloud systemd[1]: Started Open vSwitch Database Unit.
Jun 14 18:46:51 undercloud systemd[1]: Starting Open vSwitch Delete Transient Ports...
Jun 14 18:46:51 undercloud systemd[1]: Started Open vSwitch Delete Transient Ports.
Jun 14 18:46:51 undercloud systemd[1]: Starting Open vSwitch Forwarding Unit...
Jun 14 18:46:52 undercloud systemd[1]: <email address hidden>: Succeeded.
Jun 14 18:46:52 undercloud systemd[1]: Started OpenSSH rsa Server Key Generation.
Jun 14 18:46:52 undercloud systemd[1]: Reached target sshd-keygen.target.
Jun 14 18:46:52 undercloud kernel: openvswitch: Open vSwitch switching datapath
Jun 14 18:46:52 undercloud ovs-ctl[1138]: Inserting openvswitch module [ OK ]
Jun 14 18:46:52 undercloud ovs-ctl[1138]: Starting ovs-vswitchd [ OK ]
Jun 14 18:46:52 undercloud ovs-vsctl[1201]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait add Open_vSwitch . external-ids hostname=undercloud
Jun 14 18:46:52 undercloud ovs-ctl[1138]: Enabling remote OVSDB managers [ OK ]
Jun 14 18:46:52 undercloud systemd[1]: Started Open vSwitch Forwarding Unit.
Jun 14 18:46:52 undercloud systemd[1]: Starting Open vSwitch...
Jun 14 18:46:52 undercloud systemd[1]: Started Open vSwitch.
Jun 14 18:46:52 undercloud systemd[1]: Starting Network Manager...
Jun 14 18:46:52 undercloud NetworkManager[1205]: <info> [1623696412.1999] NetworkManager (version 1.32.0-0.4.el8) is starting... (for the first time)
Jun 14 18:46:52 undercloud NetworkManager[1205]: <info> [1623696412.2003] Read config: /etc/NetworkManager/NetworkManager.conf (lib: 00-server.conf)
Jun 14 18:46:52 undercloud systemd[1]: Started Network Manager.
Jun 14 18:46:52 undercloud NetworkManager[1205]: <info> [1623696412.2031] bus-manager: acquired D-Bus service "org.freedesktop.NetworkManager"
Jun 14 18:46:52 undercloud systemd[1]: Starting Network Manager Wait Online...
Jun 14 18:46:52 undercloud NetworkManager[1205]: <info> [1623696412.2115] manager[0x55d647c9d000]: monitoring kernel firmware directory '/lib/firmware'.
Jun 14 18:46:52 undercloud dbus-daemon[983]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.12' (uid=0 pid=1205 comm="/usr/sbin/NetworkManager --no-daemon " label="system_u:system_r:NetworkManager_t:s0")
Jun 14 18:46:52 undercloud systemd[1]: Starting Hostname Service...
Jun 14 18:46:52 undercloud dbus-daemon[983]: [system] Successfully activated service 'org.freedesktop.hostname1'
Jun 14 18:46:52 undercloud systemd[1]: Started Hostname Service.
Jun 14 18:46:52 undercloud NetworkManager[1205]: <info> [1623696412.3372] hostname: hostname: using hostnamed
Jun 14 18:46:52 undercloud NetworkManager[1205]: <info> [1623696412.3373] hostname: hostname changed from (none) to "undercloud"
Jun 14 18:46:52 undercloud NetworkManager[1205]: <info> [1623696412.3376] dns-mgr[0x55d647c7f250]: init: dns=default,systemd-resolved rc-manager=symlink
Jun 14 18:46:52 undercloud NetworkManager[1205]: <info> [1623696412.3457] Loaded device plugin: NMOvsFactory (/usr/lib64/NetworkManager/1.32.0-0.4.el8/libnm-device-plugin-ovs.so)
Jun 14 18:46:52 undercloud NetworkManager[1205]: <info> [1623696412.3482] Loaded device plugin: NMTeamFactory (/usr/lib64/NetworkManager/1.32.0-0.4.el8/libnm-device-plugin-team.so)
Jun 14 18:46:52 undercloud NetworkManager[1205]: <info> [1623696412.3483] manager: rfkill: Wi-Fi enabled by radio killswitch; enabled by state file
Jun 14 18:46:52 undercloud NetworkManager[1205]: <info> [1623696412.3483] manager: rfkill: WWAN enabled by radio killswitch; enabled by state file
Jun 14 18:46:52 undercloud NetworkManager[1205]: <info> [1623696412.3484] manager: Networking is enabled by state file
Jun 14 18:46:52 undercloud NetworkManager[1205]: <info> [1623696412.3484] dhcp-init: Using DHCP client 'internal'
Jun 14 18:46:52 undercloud dbus-daemon[983]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service' requested by ':1.12' (uid=0 pid=1205 comm="/usr/sbin/NetworkManager --no-daemon " label="system_u:system_r:NetworkManager_t:s0")
Jun 14 18:46:52 undercloud NetworkManager[1205]: <info> [1623696412.3509] settings: Loaded settings plugin: ifcfg-rh ("/usr/lib64/NetworkManager/1.32.0-0.4.el8/libnm-settings-plugin-ifcfg-rh.so")
Jun 14 18:46:52 undercloud NetworkManager[1205]: <info> [1623696412.3510] settings: Loaded settings plugin: keyfile (internal)
Jun 14 18:46:52 undercloud systemd[1]: Starting Network Manager Script Dispatcher Service...
Jun 14 18:46:52 undercloud NetworkManager[1205]: <info> [1623696412.3527] device (lo): carrier: link connected
Jun 14 18:46:52 undercloud NetworkManager[1205]: <info> [1623696412.3530] manager: (lo): new Generic device (/org/freedesktop/NetworkManager/Devices/1)
Jun 14 18:46:52 undercloud NetworkManager[1205]: <info> [1623696412.3540] manager: (eth0): new Ethernet device (/org/freedesktop/NetworkManager/Devices/2)
Jun 14 18:46:52 undercloud NetworkManager[1205]: <info> [1623696412.3547] device (eth0): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'external')
Jun 14 18:46:52 undercloud kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Jun 14 18:46:52 undercloud NetworkManager[1205]: <info> [1623696412.3575] device (eth0): carrier: link connected
Jun 14 18:46:52 undercloud NetworkManager[1205]: <info> [1623696412.3610] manager: (eth1): new Ethernet device (/org/freedesktop/NetworkManager/Devices/3)
Jun 14 18:46:52 undercloud NetworkManager[1205]: <info> [1623696412.3616] device (eth1): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'external')
Jun 14 18:46:52 undercloud kernel: IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
Jun 14 18:46:52 undercloud NetworkManager[1205]: <info> [1623696412.3643] device (eth1): carrier: link connected
Jun 14 18:46:52 undercloud NetworkManager[1205]: <info> [1623696412.3681] device (eth0): state change: unavailable -> disconnected (reason 'none', sys-iface-state: 'managed')
Jun 14 18:46:52 undercloud NetworkManager[1205]: <info> [1623696412.3697] device (eth1): state change: unavailable -> disconnected (reason 'none', sys-iface-state: 'managed')
Jun 14 18:46:52 undercloud NetworkManager[1205]: <info> [1623696412.3714] manager: startup complete
Jun 14 18:46:52 undercloud systemd[1]: Started Network Manager Wait Online.
Jun 14 18:46:52 undercloud systemd[1]: Starting LSB: Bring up/down networking...
Jun 14 18:46:52 undercloud dbus-daemon[983]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Jun 14 18:46:52 undercloud systemd[1]: Started Network Manager Script Dispatcher Service.
Jun 14 18:46:52 undercloud network[1218]: WARN : [network] You are using 'network' service provided by 'network-scripts', which are now deprecated.
Jun 14 18:46:52 undercloud network[1239]: You are using 'network' service provided by 'network-scripts', which are now deprecated.
Jun 14 18:46:52 undercloud network[1218]: WARN : [network] 'network-scripts' will be removed in one of the next major releases of RHEL.
Jun 14 18:46:52 undercloud network[1240]: 'network-scripts' will be removed in one of the next major releases of RHEL.
Jun 14 18:46:52 undercloud network[1218]: WARN : [network] It is advised to switch to 'NetworkManager' instead for network management.
Jun 14 18:46:52 undercloud network[1241]: It is advised to switch to 'NetworkManager' instead for network management.
Jun 14 18:46:52 undercloud NetworkManager[1205]: <info> [1623696412.5179] audit: op="connections-reload" pid=1272 uid=0 result="success"

On master-tripleo-ci branch:
[root@undercloud ~]# dnf history info 16
Transaction ID : 16
Begin time : Sun 13 Jun 2021 04:45:15 PM UTC
Begin rpmdb : 961:a8b8e539792aec96fac1442e4e18fd16b78d2d96
End time : Sun 13 Jun 2021 04:45:16 PM UTC (1 seconds)
End rpmdb : 978:58055238ae455e01a718124030b91f16b57f6671
User : <stack>
Return-Code : Success
Releasever : 8
Command Line : -v -y install python3-psutil python3-debtcollector sos device-mapper-multipath openstack-heat-agents os-net-config jq python3-dbus
Comment :
Packages Altered:
    Install openstack-heat-agents-2.3.0-0.20210331040139.b4d249a.el8.noarch @delorean-component-tripleo
    Install os-net-config-14.2.0-0.20210517154759.352227b.el8.noarch @delorean-component-tripleo
    Install python3-heat-agent-ansible-2.3.0-0.20210331040139.b4d249a.el8.noarch @delorean-component-tripleo
    Install python3-heat-agent-apply-config-2.3.0-0.20210331040139.b4d249a.el8.noarch @delorean-component-tripleo
    Install python3-heat-agent-docker-cmd-2.3.0-0.20210331040139.b4d249a.el8.noarch @delorean-component-tripleo
    Install python3-heat-agent-hiera-2.3.0-0.20210331040139.b4d249a.el8.noarch @delorean-component-tripleo
    Install python3-heat-agent-json-file-2.3.0-0.20210331040139.b4d249a.el8.noarch @delorean-component-tripleo
    Install python3-heat-agent-puppet-2.3.0-0.20210331040139.b4d249a.el8.noarch @delorean-component-tripleo
    Install jq-1.5-12.el8.x86_64 @appstream
    Install nispor-1.1.0-1.el8.x86_64 @appstream
    Install nmstate-1.1.0-0.1.el8.noarch @appstream
    Install oniguruma-6.8.2-2.el8.x86_64 @appstream
    Install python3-libnmstate-1.1.0-0.1.el8.noarch @appstream
    Install python3-nispor-1.1.0-1.el8.noarch @appstream
    Install NetworkManager-config-server-1:1.32.0-0.4.el8.noarch @baseos
    Install NetworkManager-ovs-1:1.32.0-0.4.el8.x86_64 @baseos
    Install python3-varlink-29.0.0-1.el8.noarch @baseos

On master branch, where it worked:
[root@undercloud ~]# dnf history info 16
Transaction ID : 16
Begin time : Thu 20 May 2021 12:51:50 AM UTC
Begin rpmdb : 965:763e85b8a2c148cb5750cbe4000a91ef3f4232f1
End time : Thu 20 May 2021 12:51:50 AM UTC (0 seconds)
End rpmdb : 975:55ce01b3de9a38384841ed40b4d026259ea7d44f
User : <stack>
Return-Code : Success
Releasever : 8
Command Line : -v -y install python3-psutil python3-debtcollector sos device-mapper-multipath openstack-heat-agents os-net-config jq python3-dbus
Comment :
Packages Altered:
    Install openstack-heat-agents-2.3.0-0.20210331040139.b4d249a.el8.noarch @delorean-component-tripleo
    Install os-net-config-14.2.0-0.20210510190903.352227b.el8.noarch @delorean-component-tripleo
    Install python3-heat-agent-ansible-2.3.0-0.20210331040139.b4d249a.el8.noarch @delorean-component-tripleo
    Install python3-heat-agent-apply-config-2.3.0-0.20210331040139.b4d249a.el8.noarch @delorean-component-tripleo
    Install python3-heat-agent-docker-cmd-2.3.0-0.20210331040139.b4d249a.el8.noarch @delorean-component-tripleo
    Install python3-heat-agent-hiera-2.3.0-0.20210331040139.b4d249a.el8.noarch @delorean-component-tripleo
    Install python3-heat-agent-json-file-2.3.0-0.20210331040139.b4d249a.el8.noarch @delorean-component-tripleo
    Install python3-heat-agent-puppet-2.3.0-0.20210331040139.b4d249a.el8.noarch @delorean-component-tripleo
    Install jq-1.5-12.el8.x86_64 @appstream
    Install oniguruma-6.8.2-2.el8.x86_64 @appstream

WORKAROUND:
===========

Create a undercloud-prepare-script:
cat >~/undercloud_prepare_script.sh<<EOFF
#!/bin/bash
cat >/etc/sysconfig/network-scripts/ifcfg-eth0<<EOF
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=dhcp
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME="eth0"
DEVICE=eth0
ONBOOT=yes
EOF
EOFF

Execute:
/quickstart.sh -X -n -R master-tripleo-ci --tags all -T none --extra-vars undercloud_disk=70 --extra-vars undercloud_customize_script=~/undercloud_prepare_script.sh --nodes config/nodes/3ctlr_2comp_3ceph.yml -p quickstart.yml 127.0.0.2

[yatin (yatinkarel)]

Also hitting similar issue in CI:- https://ci.centos.org/job/tripleo-quickstart-promote-master-current-tripleo-delorean-minimal/927/consoleText
https://ci.centos.org/job/tripleo-quickstart-promote-master-current-tripleo-delorean-minimal/926/consoleText

[yatin (yatinkarel)]

https://review.rdoproject.org/r/c/openstack/os-net-config-distgit/+/33475 added NetworkManager and other packages which are additional installed in bug description. If that's causing it can try disabling NetworkManager in image.

[yatin (yatinkarel)]

Also it's hitting phase 1 promotion jobs, so adding promotion-blocker tag.

[yatin (yatinkarel)]

<< https://review.rdoproject.org/r/c/openstack/os-net-config-distgit/+/33475 added NetworkManager and other packages which are additional installed in bug description. If that's causing it can try disabling NetworkManager in image.

Tried https://github.com/karelyatin/tripleo-quickstart/commit/51268e65871c94ebf47145214472c08e3b280209 but doesn't seem to work, so just those new packages alone not looks causing the issue, https://review.opendev.org/q/I3b8cb5f8c413c109da1228878db9cff4d439273b might have caused it.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/799574

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart/+/799574
Committed: https://opendev.org/openstack/tripleo-quickstart/commit/8148dc04b4a928a3af0fc87327dc33ea59fe29d1
Submitter: "Zuul (22348)"
Branch: master

commit 8148dc04b4a928a3af0fc87327dc33ea59fe29d1
Author: yatinkarel <email address hidden>
Date: Tue Jul 6 13:01:22 2021 +0530

    Ensure Network config is present for undercloud vm

    With [1] ifcfg-eth* Network configs are removed from
    overcloud images, and since for libvirt deployments
    when using overcloud_as_undercloud=true, overcloud images
    are used and undercloud vm needs the network config for
    connectivity, let's ensure it exists for such case.

    [1] https://review.opendev.org/q/I3b8cb5f8c413c109da1228878db9cff4d439273b

    Closes-Bug: #1932019
    Change-Id: Ieeb6243903b813a114c66c2bb20df8321b813762

[wes hayutin (weshayutin)]

06:14:02 TASK [setup/undercloud : Get undercloud vm ip address] *************************
06:14:02 task path: /home/jenkins/workspace/tripleo-quickstart-promote-master-current-tripleo-delorean-minimal/tripleo-quickstart/roles/libvirt/setup/undercloud/tasks/main.yml:376
06:14:02 Wednesday 07 July 2021 12:14:02 +0000 (0:00:01.431) 0:32:18.811 ********
06:14:04 FAILED - RETRYING: Get undercloud vm ip address (20 retries left).
06:14:15 FAILED - RETRYING: Get undercloud vm ip address (19 retries left).
06:14:26 changed: [172.19.3.74] => {
06:14:26 "attempts": 3,
06:14:26 "changed": true,
06:14:26 "rc": 0
06:14:26 }
06:14:26
06:14:26 STDOUT:
06:14:26
06:14:26 192.168.23.19

https://ci.centos.org/job/tripleo-quickstart-promote-master-current-tripleo-delorean-minimal/937/console

[Gael Chamoulaud (gael-chamoulaud)]

I am still hitting this issue.

TASK [setup/undercloud : Start undercloud vm] **************************************************************************************************************************************************************************************************************************************************************************
task path: /home/gchamoul/DEV/tripleo-quickstart/roles/libvirt/setup/undercloud/tasks/main.yml:364
Tuesday 20 July 2021 11:49:07 +0200 (0:00:00.338) 0:21:11.646 **********
changed: [krav.usersys.redhat.com] => {
    "changed": true
}

TASK [setup/undercloud : Get undercloud vm ip address] *****************************************************************************************************************************************************************************************************************************************************************
task path: /home/gchamoul/DEV/tripleo-quickstart/roles/libvirt/setup/undercloud/tasks/main.yml:376
Tuesday 20 July 2021 11:49:08 +0200 (0:00:00.804) 0:21:12.450 **********
FAILED - RETRYING: Get undercloud vm ip address (20 retries left).
FAILED - RETRYING: Get undercloud vm ip address (19 retries left).
FAILED - RETRYING: Get undercloud vm ip address (18 retries left).
FAILED - RETRYING: Get undercloud vm ip address (17 retries left).
FAILED - RETRYING: Get undercloud vm ip address (16 retries left).
FAILED - RETRYING: Get undercloud vm ip address (15 retries left).
FAILED - RETRYING: Get undercloud vm ip address (14 retries left).
FAILED - RETRYING: Get undercloud vm ip address (13 retries left).
FAILED - RETRYING: Get undercloud vm ip address (12 retries left).
FAILED - RETRYING: Get undercloud vm ip address (11 retries left).
FAILED - RETRYING: Get undercloud vm ip address (10 retries left).
FAILED - RETRYING: Get undercloud vm ip address (9 retries left).
FAILED - RETRYING: Get undercloud vm ip address (8 retries left).
FAILED - RETRYING: Get undercloud vm ip address (7 retries left).
FAILED - RETRYING: Get undercloud vm ip address (6 retries left).
FAILED - RETRYING: Get undercloud vm ip address (5 retries left).
FAILED - RETRYING: Get undercloud vm ip address (4 retries left).
FAILED - RETRYING: Get undercloud vm ip address (3 retries left).
FAILED - RETRYING: Get undercloud vm ip address (2 retries left).
FAILED - RETRYING: Get undercloud vm ip address (1 retries left).
fatal: [krav.usersys.redhat.com]: FAILED! => {
    "attempts": 20,
    "changed": true,
    "rc": 1
}

STDOUT:

undercloud ip is not available

STDERR:

Connection to krav.usersys.redhat.com closed.

MSG:

non-zero return code

PLAY RECAP *************************************************************************************************************************************************************************************************************************************************************************************************************
127.0.0.1 : ok=12 changed=5 unreachable=0 failed=0 skipped=28 rescued=0 ignored=0
krav.usersys.redhat.com : ok=166 changed=82 unreachable=0 failed=1 skipped=111 rescued=0 ignored=11

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/830870