master/wallaby undercloud install failure unable to start network service

Bug #1910107 reported by wes hayutin
26
This bug affects 3 people
Affects Status Importance Assigned to Milestone
tripleo
Critical
Unassigned

Bug Description

http://artifacts.ci.centos.org/rdo/jenkins-tripleo-quickstart-promote-master-current-tripleo-delorean-minimal-863/undercloud/var/log/extra/journal.txt

Jan 03 08:22:41 undercloud.localdomain python3[21945]: ansible-copy Invoked with dest=/var/lib/tripleo-config/os-net-config.returncode src=/root/.ansible/tmp/ansible-tmp-1609662160.5444527-21930-271317039721195/source _original_basename=tmpszhyceb7 follow=False checksum=b6589fc6ab0dc82cf12099d1c2d40ab994e8410c backup=False force=True content=NOT_LOGGING_PARAMETER validate=None directory_mode=None remote_src=None local_follow=None mode=None owner=None group=None seuser=None serole=None selevel=None setype=None attributes=None regexp=None delimiter=None unsafe_writes=None
Jan 03 08:22:41 undercloud.localdomain ansible-async_wrapper.py[19294]: 19295 still running (6785)
Jan 03 08:22:41 undercloud.localdomain python3[21954]: ansible-systemd Invoked with name=network enabled=True state=started daemon_reload=False daemon_reexec=False no_block=False force=None masked=None user=None scope=None
Jan 03 08:22:42 undercloud.localdomain systemd[1]: Starting LSB: Bring up/down networking...
Jan 03 08:22:42 undercloud.localdomain network[21962]: WARN : [network] You are using 'network' service provided by 'network-scripts', which are now deprecated.
Jan 03 08:22:42 undercloud.localdomain network[21975]: You are using 'network' service provided by 'network-scripts', which are now deprecated.
Jan 03 08:22:42 undercloud.localdomain network[21962]: WARN : [network] 'network-scripts' will be removed in one of the next major releases of RHEL.
Jan 03 08:22:42 undercloud.localdomain network[21976]: 'network-scripts' will be removed in one of the next major releases of RHEL.
Jan 03 08:22:42 undercloud.localdomain network[21962]: WARN : [network] It is advised to switch to 'NetworkManager' instead for network management.
Jan 03 08:22:42 undercloud.localdomain network[21977]: It is advised to switch to 'NetworkManager' instead for network management.
Jan 03 08:22:42 undercloud.localdomain NetworkManager[1019]: <info> [1609662162.2562] audit: op="connections-reload" pid=22012 uid=0 result="success"
Jan 03 08:22:42 undercloud.localdomain network[21962]: Bringing up loopback interface: [ OK ]
Jan 03 08:22:42 undercloud.localdomain NetworkManager[1019]: <info> [1609662162.4981] audit: op="connections-load" args="/etc/sysconfig/network-scripts/ifcfg-br-ctlplane" pid=22108 uid=0 result="success"
Jan 03 08:22:42 undercloud.localdomain NetworkManager[1019]: <info> [1609662162.5303] audit: op="connections-load" args="/etc/sysconfig/network-scripts/ifcfg-br-ctlplane" pid=22116 uid=0 result="success"
Jan 03 08:22:42 undercloud.localdomain network[21962]: Bringing up interface br-ctlplane: [ OK ]
Jan 03 08:22:42 undercloud.localdomain NetworkManager[1019]: <info> [1609662162.5876] audit: op="connections-load" args="/etc/sysconfig/network-scripts/ifcfg-ens3-1" pid=22133 uid=0 result="success"
Jan 03 08:22:42 undercloud.localdomain NetworkManager[1019]: <info> [1609662162.6513] agent-manager: agent[d5bac03786873671,:1.735/nmcli-connect/0]: agent registered
Jan 03 08:22:42 undercloud.localdomain NetworkManager[1019]: <info> [1609662162.6527] audit: op="connection-activate" uuid="42f5a973-856f-4335-8a46-709316517dc8" name="ens3" result="fail" reason="No suitable device found for this connection (device eth0 not available because profile is not compatible with device (mismatching interface name))."
Jan 03 08:22:42 undercloud.localdomain network[21962]: Bringing up interface ens3-1: Error: Connection activation failed: No suitable device found for this connection (device eth0 not available because profile is not compatible with device (mismatching interface name)).

Revision history for this message
wes hayutin (weshayutin) wrote :

needs more debug before moving to promotion blocker

Revision history for this message
wes hayutin (weshayutin) wrote :

From the undercloud install:
2021-01-04 15:35:41.181029 | 0004c2b4-fae6-9573-faa4-00000000004b | FATAL | Ensure network service is enabled | undercloud | error={"changed": false, "msg": "Unable to start service network: Job for network.service failed because the control process exited with error code.\nSee \"systemctl status network.service\" and \"journalctl -xe\" for details.\n"}
2021-01-04 15:35:41.182800 | 0004c2b4-fae6-9573-faa4-00000000004b | TIMING | tripleo_network_config : Ensure network service is enabled | undercloud | 0:01:32.190343 | 1.79s

Network related info:
https://artifacts.ci.centos.org/rdo/jenkins-tripleo-quickstart-promote-master-current-tripleo-delorean-minimal-865/undercloud/var/log/extra/network.txt

Revision history for this message
wes hayutin (weshayutin) wrote :

http://artifacts.ci.centos.org/rdo/jenkins-tripleo-quickstart-promote-master-current-tripleo-delorean-minimal-865/undercloud/etc/sysconfig/network-scripts/ifcfg-ens3-1

TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=dhcp
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens3
UUID=42f5a973-856f-4335-8a46-709316517dc8
DEVICE=ens3
ONBOOT=yes

Revision history for this message
wes hayutin (weshayutin) wrote :
tags: added: promotion-blocker
Revision history for this message
wes hayutin (weshayutin) wrote :

http://images.rdoproject.org/centos8/master/rdo_trunk/current-tripleo/

whayutin@localhost OVERCLOUD_IMAGES]$ sudo qemu-nbd --connect=/dev/nbd0 /var/tmp/OVERCLOUD_IMAGES/overcloud-full.qcow2
[whayutin@localhost OVERCLOUD_IMAGES]$ mkdir /tmp/wes
[whayutin@localhost OVERCLOUD_IMAGES]$ sudo mount /dev/nbd0 /tmp/wes
[whayutin@localhost OVERCLOUD_IMAGES]$ cd /tmp/wes
[whayutin@localhost wes]$ ls
bin boot dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var
[whayutin@localhost wes]$ cd etc/sysconfig/network-scripts/
[whayutin@localhost network-scripts]$ ls
ifcfg-ens3-1 ifdown-bnep ifdown-ipv6 ifdown-post ifdown-Team ifup ifup-eth ifup-isdn ifup-plusb ifup-sit ifup-tunnel network-functions
ifcfg-lo ifdown-eth ifdown-isdn ifdown-routes ifdown-TeamPort ifup-aliases ifup-ippp ifup-ovs ifup-post ifup-Team ifup-wireless network-functions-ipv6
ifdown ifdown-ippp ifdown-ovs ifdown-sit ifdown-tunnel ifup-bnep ifup-ipv6 ifup-plip ifup-routes ifup-TeamPort init.ipv6-global
[whayutin@localhost network-scripts]$

Revision history for this message
wes hayutin (weshayutin) wrote :

[whayutin@localhost network-scripts]$ ll
total 240
-rw-r--r--. 1 root root 278 Dec 4 10:41 ifcfg-ens3-1
-rw-r--r--. 1 root root 254 Jul 24 06:00 ifcfg-lo
-rwxr-xr-x. 1 root root 2123 Jul 24 06:00 ifdown
-rwxr-xr-x. 1 root root 646 Jul 24 06:00 ifdown-bnep
-rwxr-xr-x. 1 root root 6419 Jul 24 06:00 ifdown-eth
-rwxr-xr-x. 1 root root 769 Jul 24 06:00 ifdown-ippp
-rwxr-xr-x. 1 root root 4536 Jul 24 06:00 ifdown-ipv6
lrwxrwxrwx. 1 root root 11 Aug 4 15:42 ifdown-isdn -> ifdown-ippp
-rwxr-xr-x. 1 root root 2189 Feb 14 2020 ifdown-ovs
-rwxr-xr-x. 1 root root 2064 Jul 24 06:00 ifdown-post
-rwxr-xr-x. 1 root root 870 Jul 24 06:00 ifdown-routes
-rwxr-xr-x. 1 root root 1458 Jul 24 06:00 ifdown-sit
-rwxr-xr-x. 1 root root 1621 Jul 26 03:58 ifdown-Team

Revision history for this message
Rabi Mishra (rabi) wrote :

I've seen this issue earlier with local testing. I think it's an issue with the centos 8.3 cloud image which includes the 'ifcfg-ens3-1' script. We should probably just remove it as a workaround.

Revision history for this message
wes hayutin (weshayutin) wrote :
Revision history for this message
yatin (yatinkarel) wrote :

CentOS issue https://bugs.centos.org/view.php?id=17133, i have modified the image images.rdoproject.org/CentOS-8-x86_64-GenericCloud.qcow2 which is used in jobs to cleanup ens3 and ens3-1 network-scripts. https://review.opendev.org/c/openstack/tripleo-image-elements/+/769209 can also be taken as fresh images can also be used.

wes hayutin (weshayutin)
Changed in tripleo:
status: Triaged → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-image-elements 13.0.0

This issue was fixed in the openstack/tripleo-image-elements 13.0.0 release.

Revision history for this message
wes hayutin (weshayutin) wrote :
Changed in tripleo:
status: Fix Released → Triaged
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-image-elements (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/tripleo-image-elements/+/787615

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-image-elements (stable/ussuri)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-image-elements (stable/train)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-image-elements (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/tripleo-image-elements/+/787796
Committed: https://opendev.org/openstack/tripleo-image-elements/commit/49c16a368ee4e1f1ba99a04883f7342f964ab7a9
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 49c16a368ee4e1f1ba99a04883f7342f964ab7a9
Author: Wes Hayutin <email address hidden>
Date: Mon Jan 4 12:39:44 2021 -0700

    ensure any ens3 interface is nuked during image build

    It seems as though w/ CentOS-8.3 ens3 has crept
    back into play. Ensure it's not in the overcloud
    images. See bug for details.

    Closes-Bug: #1910107
    Change-Id: I29d632814577db0b720b07d4497983e267d46a4e
    (cherry picked from commit d072a79acf52a87333fa663059eb769009b8fb4e)

tags: added: in-stable-ussuri
tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-image-elements (stable/train)

Reviewed: https://review.opendev.org/c/openstack/tripleo-image-elements/+/787797
Committed: https://opendev.org/openstack/tripleo-image-elements/commit/c6d24542ab16019d643fa42ba7f5834e53d8d087
Submitter: "Zuul (22348)"
Branch: stable/train

commit c6d24542ab16019d643fa42ba7f5834e53d8d087
Author: Wes Hayutin <email address hidden>
Date: Mon Jan 4 12:39:44 2021 -0700

    ensure any ens3 interface is nuked during image build

    It seems as though w/ CentOS-8.3 ens3 has crept
    back into play. Ensure it's not in the overcloud
    images. See bug for details.

    Closes-Bug: #1910107
    Change-Id: I29d632814577db0b720b07d4497983e267d46a4e
    (cherry picked from commit d072a79acf52a87333fa663059eb769009b8fb4e)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-image-elements (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/tripleo-image-elements/+/787615
Committed: https://opendev.org/openstack/tripleo-image-elements/commit/375b4e87775b6b274cb9cc79e036e8bf6acda274
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 375b4e87775b6b274cb9cc79e036e8bf6acda274
Author: Wes Hayutin <email address hidden>
Date: Mon Jan 4 12:39:44 2021 -0700

    ensure any ens3 interface is nuked during image build

    It seems as though w/ CentOS-8.3 ens3 has crept
    back into play. Ensure it's not in the overcloud
    images. See bug for details.

    Closes-Bug: #1910107
    Change-Id: I29d632814577db0b720b07d4497983e267d46a4e
    (cherry picked from commit d072a79acf52a87333fa663059eb769009b8fb4e)

tags: added: in-stable-victoria
wes hayutin (weshayutin)
Changed in tripleo:
status: Triaged → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-image-elements 12.2.1

This issue was fixed in the openstack/tripleo-image-elements 12.2.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-image-elements 10.6.3

This issue was fixed in the openstack/tripleo-image-elements 10.6.3 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-image-elements 12.0.2

This issue was fixed in the openstack/tripleo-image-elements 12.0.2 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.