master/wallaby undercloud install failure unable to start network service

[wes hayutin]

http://artifacts.ci.centos.org/rdo/jenkins-tripleo-quickstart-promote-master-current-tripleo-delorean-minimal-863/undercloud/var/log/extra/journal.txt

Jan 03 08:22:41 undercloud.localdomain python3[21945]: ansible-copy Invoked with dest=/var/lib/tripleo-config/os-net-config.returncode src=/root/.ansible/tmp/ansible-tmp-1609662160.5444527-21930-271317039721195/source _original_basename=tmpszhyceb7 follow=False checksum=b6589fc6ab0dc82cf12099d1c2d40ab994e8410c backup=False force=True content=NOT_LOGGING_PARAMETER validate=None directory_mode=None remote_src=None local_follow=None mode=None owner=None group=None seuser=None serole=None selevel=None setype=None attributes=None regexp=None delimiter=None unsafe_writes=None
Jan 03 08:22:41 undercloud.localdomain ansible-async_wrapper.py[19294]: 19295 still running (6785)
Jan 03 08:22:41 undercloud.localdomain python3[21954]: ansible-systemd Invoked with name=network enabled=True state=started daemon_reload=False daemon_reexec=False no_block=False force=None masked=None user=None scope=None
Jan 03 08:22:42 undercloud.localdomain systemd[1]: Starting LSB: Bring up/down networking...
Jan 03 08:22:42 undercloud.localdomain network[21962]: WARN : [network] You are using 'network' service provided by 'network-scripts', which are now deprecated.
Jan 03 08:22:42 undercloud.localdomain network[21975]: You are using 'network' service provided by 'network-scripts', which are now deprecated.
Jan 03 08:22:42 undercloud.localdomain network[21962]: WARN : [network] 'network-scripts' will be removed in one of the next major releases of RHEL.
Jan 03 08:22:42 undercloud.localdomain network[21976]: 'network-scripts' will be removed in one of the next major releases of RHEL.
Jan 03 08:22:42 undercloud.localdomain network[21962]: WARN : [network] It is advised to switch to 'NetworkManager' instead for network management.
Jan 03 08:22:42 undercloud.localdomain network[21977]: It is advised to switch to 'NetworkManager' instead for network management.
Jan 03 08:22:42 undercloud.localdomain NetworkManager[1019]: <info> [1609662162.2562] audit: op="connections-reload" pid=22012 uid=0 result="success"
Jan 03 08:22:42 undercloud.localdomain network[21962]: Bringing up loopback interface: [ OK ]
Jan 03 08:22:42 undercloud.localdomain NetworkManager[1019]: <info> [1609662162.4981] audit: op="connections-load" args="/etc/sysconfig/network-scripts/ifcfg-br-ctlplane" pid=22108 uid=0 result="success"
Jan 03 08:22:42 undercloud.localdomain NetworkManager[1019]: <info> [1609662162.5303] audit: op="connections-load" args="/etc/sysconfig/network-scripts/ifcfg-br-ctlplane" pid=22116 uid=0 result="success"
Jan 03 08:22:42 undercloud.localdomain network[21962]: Bringing up interface br-ctlplane: [ OK ]
Jan 03 08:22:42 undercloud.localdomain NetworkManager[1019]: <info> [1609662162.5876] audit: op="connections-load" args="/etc/sysconfig/network-scripts/ifcfg-ens3-1" pid=22133 uid=0 result="success"
Jan 03 08:22:42 undercloud.localdomain NetworkManager[1019]: <info> [1609662162.6513] agent-manager: agent[d5bac03786873671,:1.735/nmcli-connect/0]: agent registered
Jan 03 08:22:42 undercloud.localdomain NetworkManager[1019]: <info> [1609662162.6527] audit: op="connection-activate" uuid="42f5a973-856f-4335-8a46-709316517dc8" name="ens3" result="fail" reason="No suitable device found for this connection (device eth0 not available because profile is not compatible with device (mismatching interface name))."
Jan 03 08:22:42 undercloud.localdomain network[21962]: Bringing up interface ens3-1: Error: Connection activation failed: No suitable device found for this connection (device eth0 not available because profile is not compatible with device (mismatching interface name)).

[wes hayutin (weshayutin)]

needs more debug before moving to promotion blocker

[wes hayutin (weshayutin)]

From the undercloud install:
2021-01-04 15:35:41.181029 | 0004c2b4-fae6-9573-faa4-00000000004b | FATAL | Ensure network service is enabled | undercloud | error={"changed": false, "msg": "Unable to start service network: Job for network.service failed because the control process exited with error code.\nSee \"systemctl status network.service\" and \"journalctl -xe\" for details.\n"}
2021-01-04 15:35:41.182800 | 0004c2b4-fae6-9573-faa4-00000000004b | TIMING | tripleo_network_config : Ensure network service is enabled | undercloud | 0:01:32.190343 | 1.79s

Network related info:
https://artifacts.ci.centos.org/rdo/jenkins-tripleo-quickstart-promote-master-current-tripleo-delorean-minimal-865/undercloud/var/log/extra/network.txt

[wes hayutin (weshayutin)]

http://artifacts.ci.centos.org/rdo/jenkins-tripleo-quickstart-promote-master-current-tripleo-delorean-minimal-865/undercloud/etc/sysconfig/network-scripts/ifcfg-ens3-1

TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=dhcp
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens3
UUID=42f5a973-856f-4335-8a46-709316517dc8
DEVICE=ens3
ONBOOT=yes

[wes hayutin (weshayutin)]

related codesearch:

https://codesearch.opendev.org/?q=ens3&i=nope&files=&excludeFiles=&repos=openstack/tripleo-ansible,openstack/tripleo-ci,openstack/tripleo-common,openstack/tripleo-common-tempest-plugin,openstack/tripleo-docs,openstack/tripleo-ha-utils,openstack/tripleo-heat-templates,openstack/tripleo-image-elements,openstack/tripleo-ipsec,openstack/tripleo-operator-ansible,openstack/tripleo-puppet-elements,openstack/tripleo-quickstart,openstack/tripleo-quickstart-extras,openstack/tripleo-repos,openstack/tripleo-specs,openstack/tripleo-upgrade,openstack/tripleo-validations

[wes hayutin (weshayutin)]

http://images.rdoproject.org/centos8/master/rdo_trunk/current-tripleo/

whayutin@localhost OVERCLOUD_IMAGES]$ sudo qemu-nbd --connect=/dev/nbd0 /var/tmp/OVERCLOUD_IMAGES/overcloud-full.qcow2
[whayutin@localhost OVERCLOUD_IMAGES]$ mkdir /tmp/wes
[whayutin@localhost OVERCLOUD_IMAGES]$ sudo mount /dev/nbd0 /tmp/wes
[whayutin@localhost OVERCLOUD_IMAGES]$ cd /tmp/wes
[whayutin@localhost wes]$ ls
bin boot dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var
[whayutin@localhost wes]$ cd etc/sysconfig/network-scripts/
[whayutin@localhost network-scripts]$ ls
ifcfg-ens3-1 ifdown-bnep ifdown-ipv6 ifdown-post ifdown-Team ifup ifup-eth ifup-isdn ifup-plusb ifup-sit ifup-tunnel network-functions
ifcfg-lo ifdown-eth ifdown-isdn ifdown-routes ifdown-TeamPort ifup-aliases ifup-ippp ifup-ovs ifup-post ifup-Team ifup-wireless network-functions-ipv6
ifdown ifdown-ippp ifdown-ovs ifdown-sit ifdown-tunnel ifup-bnep ifup-ipv6 ifup-plip ifup-routes ifup-TeamPort init.ipv6-global
[whayutin@localhost network-scripts]$

[wes hayutin (weshayutin)]

[whayutin@localhost network-scripts]$ ll
total 240
-rw-r--r--. 1 root root 278 Dec 4 10:41 ifcfg-ens3-1
-rw-r--r--. 1 root root 254 Jul 24 06:00 ifcfg-lo
-rwxr-xr-x. 1 root root 2123 Jul 24 06:00 ifdown
-rwxr-xr-x. 1 root root 646 Jul 24 06:00 ifdown-bnep
-rwxr-xr-x. 1 root root 6419 Jul 24 06:00 ifdown-eth
-rwxr-xr-x. 1 root root 769 Jul 24 06:00 ifdown-ippp
-rwxr-xr-x. 1 root root 4536 Jul 24 06:00 ifdown-ipv6
lrwxrwxrwx. 1 root root 11 Aug 4 15:42 ifdown-isdn -> ifdown-ippp
-rwxr-xr-x. 1 root root 2189 Feb 14 2020 ifdown-ovs
-rwxr-xr-x. 1 root root 2064 Jul 24 06:00 ifdown-post
-rwxr-xr-x. 1 root root 870 Jul 24 06:00 ifdown-routes
-rwxr-xr-x. 1 root root 1458 Jul 24 06:00 ifdown-sit
-rwxr-xr-x. 1 root root 1621 Jul 26 03:58 ifdown-Team

[Rabi Mishra (rabi)]

I've seen this issue earlier with local testing. I think it's an issue with the centos 8.3 cloud image which includes the 'ifcfg-ens3-1' script. We should probably just remove it as a workaround.

[wes hayutin (weshayutin)]

https://review.opendev.org/c/openstack/tripleo-image-elements/+/769209

[yatin (yatinkarel)]

CentOS issue https://bugs.centos.org/view.php?id=17133, i have modified the image images.rdoproject.org/CentOS-8-x86_64-GenericCloud.qcow2 which is used in jobs to cleanup ens3 and ens3-1 network-scripts. https://review.opendev.org/c/openstack/tripleo-image-elements/+/769209 can also be taken as fresh images can also be used.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-image-elements 13.0.0 release.

[wes hayutin (weshayutin)]

now blocking ci.centos jobs
https://artifacts.ci.centos.org/rdo/jenkins-tripleo-quickstart-promote-ussuri-current-tripleo-delorean-minimal-169/undercloud/home/stack/undercloud_install.log

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/tripleo-image-elements/+/787615

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/tripleo-image-elements/+/787796

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/tripleo-image-elements/+/787797

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-image-elements/+/787796
Committed: https://opendev.org/openstack/tripleo-image-elements/commit/49c16a368ee4e1f1ba99a04883f7342f964ab7a9
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 49c16a368ee4e1f1ba99a04883f7342f964ab7a9
Author: Wes Hayutin <email address hidden>
Date: Mon Jan 4 12:39:44 2021 -0700

    ensure any ens3 interface is nuked during image build

    It seems as though w/ CentOS-8.3 ens3 has crept
    back into play. Ensure it's not in the overcloud
    images. See bug for details.

    Closes-Bug: #1910107
    Change-Id: I29d632814577db0b720b07d4497983e267d46a4e
    (cherry picked from commit d072a79acf52a87333fa663059eb769009b8fb4e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-image-elements/+/787797
Committed: https://opendev.org/openstack/tripleo-image-elements/commit/c6d24542ab16019d643fa42ba7f5834e53d8d087
Submitter: "Zuul (22348)"
Branch: stable/train

commit c6d24542ab16019d643fa42ba7f5834e53d8d087
Author: Wes Hayutin <email address hidden>
Date: Mon Jan 4 12:39:44 2021 -0700

    ensure any ens3 interface is nuked during image build

    It seems as though w/ CentOS-8.3 ens3 has crept
    back into play. Ensure it's not in the overcloud
    images. See bug for details.

    Closes-Bug: #1910107
    Change-Id: I29d632814577db0b720b07d4497983e267d46a4e
    (cherry picked from commit d072a79acf52a87333fa663059eb769009b8fb4e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-image-elements/+/787615
Committed: https://opendev.org/openstack/tripleo-image-elements/commit/375b4e87775b6b274cb9cc79e036e8bf6acda274
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 375b4e87775b6b274cb9cc79e036e8bf6acda274
Author: Wes Hayutin <email address hidden>
Date: Mon Jan 4 12:39:44 2021 -0700

    ensure any ens3 interface is nuked during image build

    It seems as though w/ CentOS-8.3 ens3 has crept
    back into play. Ensure it's not in the overcloud
    images. See bug for details.

    Closes-Bug: #1910107
    Change-Id: I29d632814577db0b720b07d4497983e267d46a4e
    (cherry picked from commit d072a79acf52a87333fa663059eb769009b8fb4e)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-image-elements 12.2.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-image-elements 10.6.3 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-image-elements 12.0.2 release.