network stop working after reboot overcloud nodes on tripleo-quickstart deployments

[Juan Badia Payno]

Deployed the environment with tripleo-quickstart

Starting console:
.....
....
[ OK ] Started Login Service.
[ 9.147419] cloud-init[953]: Cloud-init v. 20.3-10.el8_4.3 running 'init-local' at Wed, 16 Jun 2021 10:04:57 +0000. Up 8.64 seconds.
[ OK ] Started Initial cloud-init job (pre-networking).
[ OK ] Reached target Network (Pre).
         Starting Open vSwitch Database Unit...
[ OK ] Started Open vSwitch Database Unit.
         Starting Open vSwitch Delete Transient Ports...
[ OK ] Started Open vSwitch Delete Transient Ports.
         Starting Open vSwitch Forwarding Unit...
[ 10.744922] openvswitch: Open vSwitch switching datapath
[ 10.814128] device ovs-system entered promiscuous mode
[ 10.816412] Timeout policy base is empty
[ 10.817097] Failed to associated timeout policy `ovs_test_tp'
[ 10.846429] device br-int entered promiscuous mode
[ 10.864947] device vlan20 entered promiscuous mode
[ 10.877688] device vlan10 entered promiscuous mode
[ 10.892602] device vlan40 entered promiscuous mode
[ 10.919643] device br-ex entered promiscuous mode
[ 10.936645] device vlan50 entered promiscuous mode
[ 10.953824] device vlan30 entered promiscuous mode
[ 10.954891] device ens3 entered promiscuous mode
[ OK ] Started Open vSwitch Forwarding Unit.
         Starting Open vSwitch...
[ OK ] Started Open vSwitch.
         Starting Network Manager...
[ OK ] Started Network Manager.
         Starting Network Manager Wait Online...
         Starting Hostname Service...
[ OK ] Started Hostname Service.
         Starting Network Manager Script Dispatcher Servi[ 11.246110] IPv6: ADDRCONF(NETDEV_UP): br-int: link is not ready

[ OK ] Started Network Manager Script Dispatcher Service.
[ OK ] Started Network Manager Wait Online.
         Starting LSB: Bring up/down networking...
[** ] A start job is running for LSB: Bri…up/down networking (17s / 5min 4s)[ 23.973513] IPv4: martian source 172.16.2.154 from 172.16.2.25, on dev vlan20
[ 23.978021] ll header: 00000000: ff ff ff ff ff ff 9e 0a 03 f8 3b 32 08 06 ..........;2..
[* ] A start job is running for LSB: Bri…up/down networking (17s / 5min 4s)[ 24.522446] IPv4: martian source 172.16.2.154 from 172.16.2.141, on dev vlan20
[ 24.523794] ll header: 00000000: ff ff ff ff ff ff 66 3d 12 9c 46 fe 08 06 ......f=..F...
[** ] A start job is running for LSB: Bri…up/down networking (18s / 5min 4s)[ 24.997535] IPv4: martian source 172.16.2.154 from 172.16.2.25, on dev vlan20
[ 24.998919] ll header: 00000000: ff ff ff ff ff ff 9e 0a 03 f8 3b 32 08 06 ..........;2..
[*** ] A start job is running for LSB: Bri…up/down networking (18s / 5min 4s)[ 25.546878] IPv4: martian source 172.16.2.154 from 172.16.2.141, on dev vlan20
[ 25.548312] ll header: 00000000: ff ff ff ff ff ff 66 3d 12 9c 46 fe 08 06 ......f=..F...
[ *** ] A start job is running for LSB: Bri…up/down networking (19s / 5min 4s)[ 26.021439] IPv4: martian source 172.16.2.154 from 172.16.2.25, on dev vlan20
[ 26.022901] ll header: 00000000: ff ff ff ff ff ff 9e 0a 03 f8 3b 32 08 06 ..........;2..
[ *** ] A start job is running for LSB: Bri…up/down networking (20s / 5min 4s)[ 26.570483] IPv4: martian source 172.16.2.154 from 172.16.2.141, on dev vlan20
[ 26.571978] ll header: 00000000: ff ff ff ff ff ff 66 3d 12 9c 46 fe 08 06 ......f=..F...
[ 27.045354] IPv4: martian source 172.16.2.154 from 172.16.2.25, on dev vlan20
[ 27.047131] ll header: 00000000: ff ff ff ff ff ff 9e 0a 03 f8 3b 32 08 06 ..........;2..
[ **] A start job is running for LSB: Bri…up/down networking (21s / 5min 4s)[ 28.070289] IPv4: martian source 172.16.1.133 from 172.16.1.222, on dev vlan30
[ 28.072105] ll header: 00000000: ff ff ff ff ff ff 8e 5d 4a 9a 63 1a 08 06 .......]J.c...
[ *] A start job is running for LSB: Bri…up/down networking (21s / 5min 4s)[ 28.298578] IPv4: martian source 172.16.1.133 from 172.16.1.79, on dev vlan30
[ 28.300366] ll header: 00000000: ff ff ff ff ff ff 4a be 15 e8 f6 2f 08 06 ......J..../..
[ **] A start job is running for LSB: Bri…up/down networking (22s / 5min 4s)[ 29.093469] IPv4: martian source 172.16.1.133 from 172.16.1.222, on dev vlan30
[ 29.095082] ll header: 00000000: ff ff ff ff ff ff 8e 5d 4a 9a 63 1a 08 06 .......]J.c...
[ 29.323291] IPv4: martian source 172.16.1.133 from 172.16.1.79, on dev vlan30
[ 29.324716] ll header: 00000000: ff ff ff ff ff ff 4a be 15 e8 f6 2f 08 06 ......J..../..
[ *** ] A start job is running for LSB: Bri…up/down networking (23s / 5min 4s)[ 30.117440] IPv4: martian source 172.16.1.133 from 172.16.1.222, on dev vlan30
[ 30.119352] ll header: 00000000: ff ff ff ff ff ff 8e 5d 4a 9a 63 1a 08 06 .......]J.c...
[ *** ] A start job is running for LSB: Bri…up/down networking (24s / 5min 4s)[ 31.141234] IPv4: martian source 172.16.1.133 from 172.16.1.222, on dev vlan30
[ 31.142676] ll header: 00000000: ff ff ff ff ff ff 8e 5d 4a 9a 63 1a 08 06 .......]J.c...
[*** ] A start job is running for LSB: Bri…up/down networking (24s / 5min 4s)[ 31.254367] IPv4: martian source 172.16.1.133 from 172.16.1.79, on dev vlan30
[ 31.255779] ll header: 00000000: ff ff ff ff ff ff 4a be 15 e8 f6 2f 08 06 ......J..../..
[ OK ] Started LSB: Bring up/down networking.
         Starting Initial cloud-init job (metadata service crawler)...
[ OK ] Reached target Network.
         Starting Dynamic System Tuning Daemon...
         Starting Enable periodic update of entitlement certificates....
         Starting Dynamic Login...
         Starting Neutron cleanup on startup...
         Starting GSSAPI Proxy Daemon...
[ OK ] Started Enable periodic update of entitlement certificates..
[ OK ] Started GSSAPI Proxy Daemon.
[ OK ] Started Dynamic Login.
[ OK ] Reached target NFS client services.
[ OK ] Reached target Remote File Systems (Pre).
[ OK ] Reached target Remote File Systems.
[ OK ] Started Neutron cleanup on startup.
[ OK ] Started Dynamic System Tuning Daemon.
[ 40.984742] cloud-init[2163]: Cloud-init v. 20.3-10.el8_4.3 running 'init' at Wed, 16 Jun 2021 10:05:29 +0000. Up 40.83 seconds.
[ 40.986900] cloud-init[2163]: ci-info: +++++++++++++++++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++++++++++++++++
[ 40.989063] cloud-init[2163]: ci-info: +------------+-------+------------------------------+---------------+--------+-------------------+
[ 40.991107] cloud-init[2163]: ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
[ 40.993225] cloud-init[2163]: ci-info: +------------+-------+------------------------------+---------------+--------+-------------------+
[ 40.995345] cloud-init[2163]: ci-info: | br-ex | True | 192.168.24.12 | 255.255.255.0 | global | 00:80:64:5e:09:bd |
[ 40.997354] cloud-init[2163]: ci-info: | br-ex | True | fe80::280:64ff:fe5e:9bd/64 | . | link | 00:80:64:5e:09:bd |
[ 40.999421] cloud-init[2163]: ci-info: | br-int | True | . | . | . | 3e:17:6c:8c:59:9f |
[ 41.001525] cloud-init[2163]: ci-info: | ens3 | True | fe80::280:64ff:fe5e:9bd/64 | . | link | 00:80:64:5e:09:bd |
[ 41.003576] cloud-init[2163]: ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | host | . |
[ 41.005625] cloud-init[2163]: ci-info: | lo | True | ::1/128 | . | host | . |
[ 41.007652] cloud-init[2163]: ci-info: | ovs-system | False | . | . | . | 4a:55:55:c4:fb:74 |
[ 41.009682] cloud-init[2163]: ci-info: | vlan10 | True | 10.0.0.241 | 255.255.255.0 | global | 0a:fa:34:61:6d:62 |
[ 41.011732] cloud-init[2163]: ci-info: | vlan10 | True | fe80::8fa:34ff:fe61:6d62/64 | . | link | 0a:fa:34:61:6d:62 |
[ 41.013695] cloud-init[2163]: ci-info: | vlan20 | True | 172.16.2.154 | 255.255.255.0 | global | f6:09:96:6a:13:72 |
[ 41.015795] cloud-init[2163]: ci-info: | vlan20 | True | fe80::f409:96ff:fe6a:1372/64 | . | link | f6:09:96:6a:13:72 |
[ 41.017717] cloud-init[2163]: ci-info: | vlan30 | True | 172.16.1.133 | 255.255.255.0 | global | 12:5a:c5:ed:c3:6a |
[ 41.019827] cloud-init[2163]: ci-info: | vlan30 | True | fe80::105a:c5ff:feed:c36a/64 | . | link | 12:5a:c5:ed:c3:6a |
[ 41.022118] cloud-init[2163]: ci-info: | vlan40 | True | 172.16.3.169 | 255.255.255.0 | global | 52:83:45:81:63:f8 |
[ 41.024319] cloud-init[2163]: ci-info: | vlan40 | True | fe80::5083:45ff:fe81:63f8/64 | . | link | 52:83:45:81:63:f8 |
[ 41.026534] cloud-init[2163]: ci-info: | vlan50 | True | 172.16.0.240 | 255.255.255.0 | global | 9e:26:80:e5:bb:dd |
[ 41.028691] cloud-init[2163]: ci-info: | vlan50 | True | fe80::9c26:80ff:fee5:bbdd/64 | . | link | 9e:26:80:e5:bb:dd |
[ 41.030714] cloud-init[2163]: ci-info: +------------+-------+------------------------------+---------------+--------+-------------------+
[ 41.032757] cloud-init[2163]: ci-info: ++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++
[ 41.034462] cloud-init[2163]: ci-info: +-------+--------------+----------+---------------+-----------+-------+
[ 41.036208] cloud-init[2163]: ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags |
[ 41.037892] cloud-init[2163]: ci-info: +-------+--------------+----------+---------------+-----------+-------+
[ 41.039552] cloud-init[2163]: ci-info: | 0 | 0.0.0.0 | 10.0.0.1 | 0.0.0.0 | vlan10 | UG |
[ 41.041102] cloud-init[2163]: ci-info: | 1 | 10.0.0.0 | 0.0.0.0 | 255.255.255.0 | vlan10 | U |
[ 41.042725] cloud-init[2163]: ci-info: | 2 | 172.16.0.0 | 0.0.0.0 | 255.255.255.0 | vlan50 | U |
[ 41.044329] cloud-init[2163]: ci-info: | 3 | 172.16.1.0 | 0.0.0.0 | 255.255.255.0 | vlan30 | U |
[ 41.045977] cloud-init[2163]: ci-info: | 4 | 172.16.2.0 | 0.0.0.0 | 255.255.255.0 | vlan20 | U |
[ 41.047534] cloud-init[2163]: ci-info: | 5 | 172.16.3.0 | 0.0.0.0 | 255.255.255.0 | vlan40 | U |
[ 41.049107] cloud-init[2163]: ci-info: | 6 | 192.168.24.0 | 0.0.0.0 | 255.255.255.0 | br-ex | U |
[ 41.052331] cloud-init[2163]: ci-info: +-------+--------------+----------+---------------+-----------+-------+
[ 41.053822] cloud-init[2163]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
[ 41.055075] cloud-init[2163]: ci-info: +-------+-------------+---------+-----------+-------+
[ 41.056268] cloud-init[2163]: ci-info: | Route | Destination | Gateway | Interface | Flags |
[ 41.057468] cloud-init[2163]: ci-info: +-------+-------------+---------+-----------+-------+
[ 41.058561] cloud-init[2163]: ci-info: | 10 | fe80::/64 | :: | br-ex | U |
[ 41.060245] cloud-init[2163]: ci-info: | 11 | fe80::/64 | :: | ens3 | U |
[ 41.061526] cloud-init[2163]: ci-info: | 12 | fe80::/64 | :: | vlan10 | U |
[ 41.062667] cloud-init[2163]: ci-info: | 13 | fe80::/64 | :: | vlan20 | U |
[ 41.063861] cloud-init[2163]: ci-info: | 14 | fe80::/64 | :: | vlan30 | U |
[ 41.065076] cloud-init[2163]: ci-info: | 15 | fe80::/64 | :: | vlan40 | U |
[ 41.066328] cloud-init[2163]: ci-info: | 16 | fe80::/64 | :: | vlan50 | U |
[ 41.067536] cloud-init[2163]: ci-info: | 18 | local | :: | br-ex | U |
[ 41.068745] cloud-init[2163]: ci-info: | 19 | local | :: | ens3 | U |
[ 41.069949] cloud-init[2163]: ci-info: | 20 | local | :: | vlan10 | U |
[ 41.071180] cloud-init[2163]: ci-info: | 21 | local | :: | vlan30 | U |
[ 41.072259] cloud-init[2163]: ci-info: | 22 | local | :: | vlan40 | U |
[ 41.073335] cloud-init[2163]: ci-info: | 23 | local | :: | vlan50 | U |
[ 41.074431] cloud-init[2163]: ci-info: | 24 | local | :: | vlan20 | U |
[ 41.075525] cloud-init[2163]: ci-info: | 25 | multicast | :: | br-int | U |
[ 41.076626] cloud-init[2163]: ci-info: | 26 | multicast | :: | br-ex | U |
[ 41.077711] cloud-init[2163]: ci-info: | 27 | multicast | :: | ens3 | U |
[ 41.078911] cloud-init[2163]: ci-info: | 28 | multicast | :: | vlan10 | U |
[ 41.080127] cloud-init[2163]: ci-info: | 29 | multicast | :: | vlan20 | U |
[ 41.081339] cloud-init[2163]: ci-info: | 30 | multicast | :: | vlan30 | U |
[ 41.082550] cloud-init[2163]: ci-info: | 31 | multicast | :: | vlan40 | U |
[ 41.083767] cloud-init[2163]: ci-info: | 32 | multicast | :: | vlan50 | U |
[ 41.084959] cloud-init[2163]: ci-info: +-------+-------------+---------+-----------+-------+
[ 41.086048] cloud-init[2163]: 2021-06-16 10:05:30,014 - stages.py[WARNING]: Failed to rename devices: [unknown] Error performing rename('ens3', 'br-ex') for 00:80:64:5e:09:bd, br-ex: Unexpected error while running command.
[ 41.088657] cloud-init[2163]: Command: ['ip', 'link', 'set', 'ens3', 'name', 'br-ex']
[ 41.089654] cloud-init[2163]: Exit code: 2
[ 41.090217] cloud-init[2163]: Reason: -
[ 41.090742] cloud-init[2163]: Stdout:
[ 41.091250] cloud-init[2163]: Stderr: RTNETLINK answers: File exists
[ OK ] Started Initial cloud-init job (metadata service crawler).

Everything seems ok:
[root@overcloud-controller-0 ~]# ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master ovs-system state DOWN group default qlen 1000
    link/ether 00:80:64:5e:09:bd brd ff:ff:ff:ff:ff:ff
3: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 4a:55:55:c4:fb:74 brd ff:ff:ff:ff:ff:ff
4: br-int: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 3e:17:6c:8c:59:9f brd ff:ff:ff:ff:ff:ff
5: vlan20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether f6:09:96:6a:13:72 brd ff:ff:ff:ff:ff:ff
    inet 172.16.2.154/24 brd 172.16.2.255 scope global vlan20
       valid_lft forever preferred_lft forever
    inet6 fe80::f409:96ff:fe6a:1372/64 scope link
       valid_lft forever preferred_lft forever
6: vlan10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 0a:fa:34:61:6d:62 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.241/24 brd 10.0.0.255 scope global vlan10
       valid_lft forever preferred_lft forever
    inet6 fe80::8fa:34ff:fe61:6d62/64 scope link
       valid_lft forever preferred_lft forever
7: vlan40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 52:83:45:81:63:f8 brd ff:ff:ff:ff:ff:ff
    inet 172.16.3.169/24 brd 172.16.3.255 scope global vlan40
       valid_lft forever preferred_lft forever
    inet6 fe80::5083:45ff:fe81:63f8/64 scope link
       valid_lft forever preferred_lft forever
8: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 00:80:64:5e:09:bd brd ff:ff:ff:ff:ff:ff
    inet 192.168.24.12/24 brd 192.168.24.255 scope global br-ex
       valid_lft forever preferred_lft forever
    inet6 fe80::280:64ff:fe5e:9bd/64 scope link
       valid_lft forever preferred_lft forever
9: vlan50: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 9e:26:80:e5:bb:dd brd ff:ff:ff:ff:ff:ff
    inet 172.16.0.240/24 brd 172.16.0.255 scope global vlan50
       valid_lft forever preferred_lft forever
    inet6 fe80::9c26:80ff:fee5:bbdd/64 scope link
       valid_lft forever preferred_lft forever
10: vlan30: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 12:5a:c5:ed:c3:6a brd ff:ff:ff:ff:ff:ff
    inet 172.16.1.133/24 brd 172.16.1.255 scope global vlan30
       valid_lft forever preferred_lft forever
    inet6 fe80::105a:c5ff:feed:c36a/64 scope link
       valid_lft forever preferred_lft forever

[root@overcloud-controller-0 ~]# ovs-vsctl show
64ecb89c-ab8f-4337-be66-6fff637d8801
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
    Bridge br-ex
        fail_mode: standalone
        Port vlan30
            tag: 30
            Interface vlan30
                type: internal
        Port vlan20
            tag: 20
            Interface vlan20
                type: internal
        Port vlan50
            tag: 50
            Interface vlan50
                type: internal
        Port ens3
            Interface ens3
        Port vlan40
            tag: 40
            Interface vlan40
                type: internal
        Port vlan10
            tag: 10
            Interface vlan10
                type: internal
        Port br-ex
            Interface br-ex
                type: internal
    ovs_version: "2.13.4"
[root@overcloud-controller-0 ~]#

WORKAROUND:
 login into the server and rise the device
 - ifconfig ens3 up

[Juan Badia Payno (jbadiapa)]

[root@overcloud-controller-0 ~]# dnf history info 16
Transaction ID : 16
Begin time : Sun 13 Jun 2021 04:45:15 PM UTC
Begin rpmdb : 961:a8b8e539792aec96fac1442e4e18fd16b78d2d96
End time : Sun 13 Jun 2021 04:45:16 PM UTC (1 seconds)
End rpmdb : 978:58055238ae455e01a718124030b91f16b57f6671
User : <heat-admin>
Return-Code : Success
Releasever : 8
Command Line : -v -y install python3-psutil python3-debtcollector sos device-mapper-multipath openstack-heat-agents os-net-config jq python3-dbus
Comment :
Packages Altered:
    Install openstack-heat-agents-2.3.0-0.20210331040139.b4d249a.el8.noarch @delorean-component-tripleo
    Install os-net-config-14.2.0-0.20210517154759.352227b.el8.noarch @delorean-component-tripleo
    Install python3-heat-agent-ansible-2.3.0-0.20210331040139.b4d249a.el8.noarch @delorean-component-tripleo
    Install python3-heat-agent-apply-config-2.3.0-0.20210331040139.b4d249a.el8.noarch @delorean-component-tripleo
    Install python3-heat-agent-docker-cmd-2.3.0-0.20210331040139.b4d249a.el8.noarch @delorean-component-tripleo
    Install python3-heat-agent-hiera-2.3.0-0.20210331040139.b4d249a.el8.noarch @delorean-component-tripleo
    Install python3-heat-agent-json-file-2.3.0-0.20210331040139.b4d249a.el8.noarch @delorean-component-tripleo
    Install python3-heat-agent-puppet-2.3.0-0.20210331040139.b4d249a.el8.noarch @delorean-component-tripleo
    Install jq-1.5-12.el8.x86_64 @appstream
    Install nispor-1.1.0-1.el8.x86_64 @appstream
    Install nmstate-1.1.0-0.1.el8.noarch @appstream
    Install oniguruma-6.8.2-2.el8.x86_64 @appstream
    Install python3-libnmstate-1.1.0-0.1.el8.noarch @appstream
    Install python3-nispor-1.1.0-1.el8.noarch @appstream
    Install NetworkManager-config-server-1:1.32.0-0.4.el8.noarch @baseos
    Install NetworkManager-ovs-1:1.32.0-0.4.el8.x86_64 @baseos
    Install python3-varlink-29.0.0-1.el8.noarch @baseos

[Juan Badia Payno (jbadiapa)]

the repo:
---------
[delorean-component-tripleo]
name=delorean-tripleo-operator-ansible-c1527ca9fcd5d2a50af896e2335b0af992d7a94b
baseurl=https://trunk.rdoproject.org/centos8/component/tripleo/c1/52/c1527ca9fcd5d2a50af896e2335b0af992d7a94b_2ecfbcb5
enabled=1
gpgcheck=0
priority=20

[Cristian Le (lecris)]

Recently affected by this issue on normal tripleo deployment. `br-int` and related devices are shown down after a reboot.