Bionic networking failures after NIC reordering

[Chris Patterson]

Documented across https://bugs.launchpad.net/bugs/1958280 and https://canonical.force.com/ua/s/case/5004K00000E96qlQAB/vf-nic-not-getting-renamed-properly-for-ubuntu-2004.

Creating this bug to focus on Bionic, because it's different than 20.04+ and last week's failure https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1988119 helped me identify part of the root cause.

When NICs are renamed on boot, networkd tends to fail to configure them.

####################################
# WITHOUT THE PROPOSED SYSTEMD PATCH
####################################

cpatterson@test-ubu1804-nicrenamerepro-x1:~$ networkctl list
IDX LINK TYPE OPERATIONAL SETUP
  1 lo loopback carrier unmanaged
  2 eth0 ether routable configured
  3 eth1 ether n/a unmanaged
  4 eth2 ether routable configured
  5 eth3 ether routable configured
  6 eth4 ether routable configured
  7 eth5 ether off unmanaged
  8 eth6 ether off unmanaged
  9 eth7 ether off unmanaged

### As expected, we can see the properties are missing.

cpatterson@test-ubu1804-nicrenamerepro-x1:~$ sudo udevadm info /sys/class/net/eth7
P: /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/device:07/VMBUS:01/0022481f-69aa-0022-481f-69aa0022481f/net/eth7
E: DEVPATH=/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/device:07/VMBUS:01/0022481f-69aa-0022-481f-69aa0022481f/net/rename9
E: ID_NET_NAME_MAC=enx0022481f69aa
E: ID_OUI_FROM_DATABASE=Microsoft Corporation
E: ID_PATH=acpi-VMBUS:01
E: ID_PATH_TAG=acpi-VMBUS_01
E: IFINDEX=9
E: INTERFACE=eth1
E: SUBSYSTEM=net
E: SYSTEMD_ALIAS=/sys/subsystem/net/devices/rename9 /sys/subsystem/net/devices/eth1 /sys/subsystem/net/devices/cirename0 /sys/subsystem/net/devices/eth7
E: TAGS=:systemd:
E: USEC_INITIALIZED=11203606

### As expected, restarting networkd does not fix the issue.

cpatterson@test-ubu1804-nicrenamerepro-x1:~$ sudo systemctl restart systemd-networkd
cpatterson@test-ubu1804-nicrenamerepro-x1:~$ networkctl list
IDX LINK TYPE OPERATIONAL SETUP
  1 lo loopback carrier unmanaged
  2 eth0 ether routable configured
  3 eth1 ether off unmanaged
  4 eth2 ether routable configured
  5 eth3 ether routable configured
  6 eth4 ether routable configured
  7 eth5 ether off unmanaged
  8 eth6 ether off unmanaged
  9 eth7 ether off unmanaged

9 links listed.

####################################
# WITH THE PROPOSED SYSTEMD PATCH
####################################

I built systemd with the proposed patches in https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1988119. With these patches, networking still comes up broken, but restarting networkd does fix things.

cpatterson@test-ubu1804-nicrenamerepro-systemd55-x2:~$ networkctl list
IDX LINK TYPE OPERATIONAL SETUP
  1 lo loopback carrier unmanaged
  2 eth0 ether routable configured
  3 eth1 ether n/a unmanaged
  4 eth2 ether n/a unmanaged
  5 eth3 ether n/a unmanaged
  6 eth4 ether routable configured
  7 eth5 ether n/a unmanaged
  8 eth6 ether n/a unmanaged
  9 eth7 ether n/a unmanaged

9 links listed.

cpatterson@test-ubu1804-nicrenamerepro-systemd55-x2:~$ sudo udevadm info /sys/class/net/eth1
P: /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/device:07/VMBUS:01/0022482b-f769-0022-482b-f7690022482b/net/eth1
E: DEVPATH=/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/device:07/VMBUS:01/0022482b-f769-0022-482b-f7690022482b/net/rename3
E: ID_NET_DRIVER=hv_netvsc
E: ID_NET_LINK_FILE=/run/systemd/network/10-netplan-eth7.link
E: ID_NET_NAME=eth1
E: ID_NET_NAME_MAC=enx0022482bf769
E: ID_OUI_FROM_DATABASE=Microsoft Corporation
E: ID_PATH=acpi-VMBUS:01
E: ID_PATH_TAG=acpi-VMBUS_01
E: IFINDEX=3
E: INTERFACE=eth7
E: NM_UNMANAGED=1
E: SUBSYSTEM=net
E: SYSTEMD_ALIAS=/sys/subsystem/net/devices/rename3 /sys/subsystem/net/devices/eth7 /sys/subsystem/net/devices/eth1
E: TAGS=:systemd:
E: USEC_INITIALIZED=10280176

cpatterson@test-ubu1804-nicrenamerepro-systemd55-x2:~$ sudo systemctl restart systemd-networkd
cpatterson@test-ubu1804-nicrenamerepro-systemd55-x2:~$ networkctl list
IDX LINK TYPE OPERATIONAL SETUP
  1 lo loopback carrier unmanaged
  2 eth0 ether routable configured
  3 eth1 ether routable configured
  4 eth2 ether routable configured
  5 eth3 ether routable configured
  6 eth4 ether routable configured
  7 eth5 ether routable configured
  8 eth6 ether routable configured
  9 eth7 ether routable configured

9 links listed.

I've attached the journal logs for repro with the proposed systemd (237-3ubuntu10.55).

[Chris Patterson (cjp256)]

Reproducer script for both variants of systemd.