netplan should consider adding more udev attribute for exact matching of failover 3-netdev interfaces

[Si-Wei Liu]

[Impact]

* At present, virtual machines utilizing net_failover network interface configurations are incorrectly configured due to the reliance on the MAC address to identify specific network interfaces. When net_failover is utilized, multiple interfaces will bear the same MAC address (the net_failover master itself, as well as the interfaces subordinate to it), rendering the MAC address ineffective for unique identification of the interface. This results in incorrect naming of network interfaces from the "set-name" directive in the netplan configuration.

* The solution here is to use the interface name instead of the MAC address when the interface is a net_failover master device. Logic is added on initramfs-tools to check the device type and virtio flags to apply this change only to net_failover master devices.

[Test Case]

* The change can be tested by configuring a virtual machine with a virtio_net network device with the "failover=on" option to the "-device" option to qemu, e.g.,

-device virtio-net-pci,netdev=hostnet0,id=net0,bus=pci.0,addr=0x3,mac=00:00:17:00:18:04,failover=on

* This will set the virtio device "standby" feature bit (bit 62, counting from 0). This requires a version of qemu with support for this feature.

* When so configured, the netplan configuration generated by initramfs will not contain a "macaddress:" match directive for the network interface in question.

[Regression Potential]

* Erroneous identification of a network interface as a net_failover master device could lead to omission of a macaddress directive, causing interfaces to be incorrectly named.

[Mathieu Trudel-Lapierre (cyphermox)]

netplan is currently writing about as much as we can for the networkd/udev configs; some values we don't know how to handle at all.

Looking at this, it feels to me like there will indeed be a need to find a different data point to differentiate the interfaces, and MAC and driver are not sufficient. This might require work in udev, so I've added a task for systemd.

Could you please provide us with more information on these devices? Could you please run 'udevadm info' for the slaves as well, so we see if there are any extra fields we can use?

If we can't find anything usable already in udev, then we might need to consider modifying the kernel driver itself to expose some information that will be needed to make the difference between the devices.

This is Triaged/Undecided for netplan, since it's quite obviously a limitation, it will need work to implement, but we can't really prioritize it before knowing what data we can work with to match on the interfaces / work done on systemd/udev.

[Si-Wei Liu (siwliu)]

There's barely anything can be used to distinguish but only the name (or ifindex) itself.

root@node-1970:~# udevadm info /sys/class/net/ens3nsby
P: /devices/pci0000:00/0000:00:03.0/virtio0/net/ens3nsby
E: DEVPATH=/devices/pci0000:00/0000:00:03.0/virtio0/net/ens3nsby
E: ID_BUS=pci
E: ID_MODEL_FROM_DATABASE=Virtio network device
E: ID_MODEL_ID=0x1000
E: ID_NET_DRIVER=virtio_net
E: ID_NET_LINK_FILE=/lib/systemd/network/99-default.link
E: ID_NET_NAME_MAC=enx02001701d517
E: ID_NET_NAME_PATH=enp0s3nsby
E: ID_NET_NAME_SLOT=ens3nsby
E: ID_PATH=pci-0000:00:03.0
E: ID_PATH_TAG=pci-0000_00_03_0
E: ID_PCI_CLASS_FROM_DATABASE=Network controller
E: ID_PCI_SUBCLASS_FROM_DATABASE=Ethernet controller
E: ID_VENDOR_FROM_DATABASE=Red Hat, Inc.
E: ID_VENDOR_ID=0x1af4
E: IFINDEX=3
E: INTERFACE=ens3nsby
E: SUBSYSTEM=net
E: SYSTEMD_ALIAS=/sys/subsystem/net/devices/ens3nsby
E: TAGS=:systemd:
E: USEC_INITIALIZED=2496132
E: net.ifnames=1

root@node-1970:~# udevadm info /sys/class/net/ens3
P: /devices/pci0000:00/0000:00:03.0/virtio0/net/ens3
E: DEVPATH=/devices/pci0000:00/0000:00:03.0/virtio0/net/ens3
E: ID_BUS=pci
E: ID_MODEL_FROM_DATABASE=Virtio network device
E: ID_MODEL_ID=0x1000
E: ID_NET_DRIVER=net_failover
E: ID_NET_LINK_FILE=/lib/systemd/network/99-default.link
E: ID_NET_NAME_MAC=enx02001701d517
E: ID_NET_NAME_PATH=enp0s3
E: ID_NET_NAME_SLOT=ens3
E: ID_PATH=pci-0000:00:03.0
E: ID_PATH_TAG=pci-0000_00_03_0
E: ID_PCI_CLASS_FROM_DATABASE=Network controller
E: ID_PCI_SUBCLASS_FROM_DATABASE=Ethernet controller
E: ID_VENDOR_FROM_DATABASE=Red Hat, Inc.
E: ID_VENDOR_ID=0x1af4
E: IFINDEX=2
E: INTERFACE=ens3
E: SUBSYSTEM=net
E: SYSTEMD_ALIAS=/sys/subsystem/net/devices/ens3
E: TAGS=:systemd:
E: USEC_INITIALIZED=2472184
E: net.ifnames=1

Below is a script we are using, which gets executed early in udev rules, to identify the role of each interface for 3-netdev.

# cat /lib/udev/detect_failover
#!/bin/sh -e
#
# Copyright (C) 2019 Oracle Corporation.
#
# This program is free software; you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the
# Free Software Foundation version 2 of the License.
#
# This script is run to detect the role of failover network device
# so as to define naming rules based on properties of the device.

get_drvname() {
        /bin/basename `/bin/readlink ${1}/device/driver`
}

has_standby() {
        features=`cat ${1}/device/features`
        [ "${features:62:1}" = "1" ]
}

# debug, if UDEV_LOG=<debug>
if [ -n "$UDEV_LOG" ]; then
        if [ "$UDEV_LOG" -ge 7 ]; then
                set -x
        fi
fi
if [ -z "$INTERFACE" ]; then
        exit 1
fi

devpath="/sys/class/net/$INTERFACE"
if [ ! -d $devpath -o ! -d ${devpath}/device ]; then
        exit 1
fi

if [ `get_drvname $devpath` = virtio_net ] && has_standby $devpath; then
        if [ -d ${devpath}/master ]; then
                echo 3 # Failover standby
        else
                echo 1 # Failover master
        fi
elif [ -d ${devpath}/master ] && [ `get_drvname ${devpath}/master` = virtio_net ] && has_standby ${devpath}/master; then
        echo 2 # Failover primary
fi
exit 0

[Si-Wei Liu (siwliu)]

Note only DHCP configuration exhibits this particular issue. The cause is that when "dhcp4" is set to "yes" in the /etc/netplan/*.yaml config file, netplan would generate a temporary per-interface yaml file under /run/netplan, which matches interface using MAC address. While I don't see this specific MAC matching when using static IP configuration.

vsbalakr@ubuntu-18:~$ cat /run/netplan/ens3.yaml
network:
  version: 2
  renderer: networkd
  ethernets:
    ens3:
      match:
        macaddress: "ba:fb:9f:12:2f:02"
      set-name: ens3
      dhcp4: true
      dhcp-identifier: mac
      critical: true
vsbalakr@ubuntu-18:~$ cat /etc/netplan/01-netcfg.yaml
# This file describes the network interfaces available on your system
# For more information, see netplan(5).
network:
  version: 2
  renderer: networkd
  ethernets:
    ens3:
      dhcp4: yes
      gateway4: 10.211.8.1
      nameservers:
              addresses: [10.211.11.1,10.209.76.197]

Which in turn gets systemd-networkd to generate these config files, also using MAC address to match interface:

vsbalakr@ubuntu-18:~$ cat /run/systemd/network/10-netplan-ens3.link
[Match]
MACAddress=ba:fb:9f:12:2f:02
OriginalName=ens3

[Link]
Name=ens3
WakeOnLan=off
vsbalakr@ubuntu-18:~$ cat /run/systemd/network/10-netplan-ens3.network
[Match]
MACAddress=ba:fb:9f:12:2f:02
Name=ens3

[Network]
DHCP=ipv4
LinkLocalAddressing=ipv6
Gateway=10.211.8.1
DNS=10.211.11.1
DNS=10.209.76.197

[DHCP]
UseMTU=true
RouteMetric=100
ClientIdentifier=mac
CriticalConnection=true

Why DHCP config needs to match MAC while static config doesn't? Is it possible not to match by MAC for DHCP, unless users explicitly ask for it?

[Mathieu Trudel-Lapierre (cyphermox)]

All of these values should be coming directly from the netplan YAML. Are all of these options required?

I see:

match:
  macaddress: xxxx # Do you need to match the interface in this case? Is it sufficient to match by name if only ens3 is being configured?

Also:

dhcp-identifier: mac # This should only be required for clients who do not have control over their DHCP server, and thus can't use the DUID to set reservations and instead must use the MAC address to get a "static" IP from DHCP. I would recommend not setting this if it can be avoided.

Also:

critical: true # This is only required for systems booting off the network to reach their root filesystem. It will control whether the DHCP IP is released / regained when systemd-networkd restarts. Again, I would recommend not setting this if it's not required; DHCP leases lifetimes should ensure you will get the same address again.

[Mathieu Trudel-Lapierre (cyphermox)]

I also forgot to mention another entry I see in some of the configs:

set-name: ens3

If you do not need to explicitly rename the interfaces yourself to a different name, I would avoid setting this at all. It *may* be being set automatically by cloud-init (if that's in use, but the configs provided do not look like generated configs from cloud-init).

I'm curious to see how this behaves with the most simple netplan configuration possible; if you still see renaming issues in that case from the kernel -- as those would be renaming done completely outside of netplan and further direct our debugging to systemd-network / systemd-udev instead.

In short; how much of the issues are you seeing when the only netplan config on the systems is /etc/netplan/01-netcfg.yaml (the one generated by debian-installer)?

Judging from the original description, I'd say you still get renaming issues, specifically any new device with the MAC is attempted to be named "ens3" because of the udev rules:

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="ba:fb:9f:12:2f:02", NAME="ens3"

This part should not vary whether there are statically assigned addresses or not; but it's also questionable whether or not it is useful to write it in the first place: systemd-udev should already be following ifnames policy to name the device appropriately (this should be true in all scenarios, this is only marginally useful for /renaming/ (when set-name: is specified) or setting MTU and such low-level options).

Finally, could you please also provide us with the output of:
sudo udevadm info /sys/class/net/ens4 (or rename4, depending on how it ends up being named).

[Si-Wei Liu (siwliu)]

Before answering your quesitions, I would wonder what do you know generates the /run/netplan/ens3.yaml file in the first place? This YAML file is not provided by us obviously, the only netplan config we have on the root disk is /etc/netplan/01-netcfg.yaml, derived from the debian installer. The YAML under /run overrides the one under /etc, for the DHCP case, and that's where the odds about the additional udev rule and various config options came from.

The DHCP server in our lab setup might have directed netroot to iSCSI so that *might* explain where the /run/netplan/ens3.yaml with tne extra match-mac, dhcp-identifier and critical options?

$ lsscsi -t
[6:0:0:0] storage iqn.2015-02.oracletest.boot:uefi,t,0x1 -
[6:0:0:1] disk iqn.2015-02.oracletest.boot:uefi,t,0x1 /dev/sda

[Mathieu Trudel-Lapierre (cyphermox)]

Yes, that does explain it.

/run/netplan/<interface>.yaml is written automatically by initramfs-tools when booting with a remote root (ie. iscsi); so this does check out: for example, 'critical' is required in that case, otherwise as soon as someone runs 'netplan apply' the network will go down and you might have filesystem errors.

[Si-Wei Liu (siwliu)]

So what's your opinion for the disposition? I don't mind moving this bug to initramfs-tools, but without having a way to identify the 3-netdev master interface in netplan yaml config, thing still don't work there. I'm not in a position to tell if it's possible for initramfs-tools to remove the "match by mac" in the /run/netplan/<interface>.yaml config file.

[Si-Wei Liu (siwliu)]

FWIW this is the 'udevadm info' output on the primary slave:

root@node-1970:~# udevadm info /sys/class/net/ens4
P: /devices/pci0000:00/0000:00:04.0/net/ens4
E: DEVPATH=/devices/pci0000:00/0000:00:04.0/net/ens4
E: ID_BUS=pci
E: ID_MODEL_FROM_DATABASE=82599 Ethernet Controller Virtual Function
E: ID_MODEL_ID=0x10ed
E: ID_NET_DRIVER=ixgbevf
E: ID_NET_LINK_FILE=/lib/systemd/network/99-default.link
E: ID_NET_NAME=ens4
E: ID_NET_NAME_MAC=enx02001701d517
E: ID_NET_NAME_PATH=enp0s4
E: ID_NET_NAME_SLOT=ens4
E: ID_PATH=pci-0000:00:04.0
E: ID_PATH_TAG=pci-0000_00_04_0
E: ID_PCI_CLASS_FROM_DATABASE=Network controller
E: ID_PCI_SUBCLASS_FROM_DATABASE=Ethernet controller
E: ID_VENDOR_FROM_DATABASE=Intel Corporation
E: ID_VENDOR_ID=0x8086
E: IFINDEX=9
E: INTERFACE=ens4
E: SUBSYSTEM=net
E: SYSTEMD_ALIAS=/sys/subsystem/net/devices/ens4 /sys/subsystem/net/devices/ens4
E: TAGS=:systemd:
E: USEC_INITIALIZED=199842925
E: net.ifnames=1

The ID_NET_DRIVER here is ixgbevf, but in general it can be driver name for any netdev including virtio_net.

[Guilherme G. Piccoli (gpiccoli)]

Although the debdiff is hereby attached, 3 bugs have fixes carried on such patch - the main work is done on LP ##1879980 (and the other LP handled in this SRU is #1879987) .

[Eric Desrochers (slashd)]

[sts-sponsor]

Sponsored in Bionic.

Thanks for your contribution.

[Mauricio Faria de Oliveira (mfo)]

Uploaded initramfs-tools to Bionic.
Attaching the updated debdiff for reference.
(Rebased on top of the more recent -updates.)

[Łukasz Zemczak (sil2100)]

Hello Si-Wei, or anyone else affected,

Accepted initramfs-tools into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/initramfs-tools/0.130ubuntu3.11 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Mauricio Faria de Oliveira (mfo)]

This has been verified successfully on Bionic by @jvosburgh
on a (complex) internal test setup for 3-netdev naming work.

[Launchpad Janitor (janitor)]

This bug was fixed in the package initramfs-tools - 0.130ubuntu3.11

---------------
initramfs-tools (0.130ubuntu3.11) bionic; urgency=medium

  [ Guilherme G. Piccoli ]
  * scripts/functions: Prevent printf error carry over if the wrong
    console is set. (LP: #1879987)
      The function _log_msg() is "void" typed, returning whatever its
      last command returns. This function is the basic building block
      for all error/warning messages in initramfs-tools. If a bad console
      is provided to kernel on command-line, printf returns error, and so
      this error is carried over in _log_msg(). Happens that checkfs()
      function has a loop that runs forever in this scenario (*if* fsck
      is not present in initramfs and "quiet" is not passed in the
      command-line). If that happens, boot is stuck and cannot progress.
      The simple fix hereby merged is to return zero on _log_msg().

  * scripts/local: Re-execute cryptroot local-block script. (LP: #1879980)
      Currently, if an encrypted rootfs is configured on top of a MD RAID1
      array and such array gets degraded (like a member is removed/failed),
      initramfs-tools cannot mount the rootfs and the boot fails. We fix
      that issue here by allowing cryptroot script to re-run on local-block
      stage, given that mdadm is able to activate a degraded array in that
      point. There is a cryptsetup counter-part for this fix, but alone the
      initramfs-tools portion is innocuous.

  [ Jay Vosburgh ]
  * scripts/functions: Change netplan render for net_failover master
    devices. (LP: #1820929)
      Modify the _render_netplan function to check for network interfaces
      that are net_failover master devices. When found, such devices are
      matched only by name, not by MAC address, as the MAC is not a unique
      identifier for the net_failover case. In the net_failover architecture,
      the MAC address is used to manage the membership of the net_failover
      interface set, thus multiple interfaces will be assigned the same MAC
      address.

 -- <email address hidden> (Guilherme G. Piccoli) Wed, 12 Aug 2020 17:12:11 -0300

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for initramfs-tools has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Si-Wei Liu (siwliu)]

I will get some time to test it this week. Are there other packages than initramfs-tools that are deem required to make net_failover work on Bionic? I recall there's some systemd-udevd level change needed not just initramfs-tools that has to be fixed.

[Jay Vosburgh (jvosburgh)]

Si-Wei,

In the test environment I'm using, the only change needed was to initramfs-tools. I suspect the udevd change you're thinking of was an alternate implementation that we did not proceed with due to the regression it introduced (that network interface names would change).

[Si-Wei Liu (siwliu)]

Jay,

The good news is that, with 0.130ubuntu3.11, initramfs-tools is now able to render a correct netplan YAML file:

# cat /run/netplan/ens3.yaml
network:
  version: 2
  renderer: networkd
  ethernets:
    ens3:
      set-name: ens3
      dhcp4: true
      dhcp-identifier: mac
      critical: true
      nameservers:
        addresses: ["100.100.100.60", "100.100.100.160"]
        search: ["domainname.com"]

However, net_failover 3-netdev interfaces are still showing incorrect names:

# ip l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens3: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 12:3f:bd:fd:bb:1e brd ff:ff:ff:ff:ff:ff
3: rename3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ens3 state UP mode DEFAULT group default qlen 1000
    link/ether 12:3f:bd:fd:bb:1e brd ff:ff:ff:ff:ff:ff
4: rename4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ens3 state UP mode DEFAULT group default qlen 1000
    link/ether 12:3f:bd:fd:bb:1e brd ff:ff:ff:ff:ff:ff

while we should expect to see something like:

# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens3: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 12:3f:bd:fd:bb:1e brd ff:ff:ff:ff:ff:ff
3: ens3nsby: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ens3 state UP mode DEFAULT group default qlen 1000
    link/ether 12:3f:bd:fd:bb:1e brd ff:ff:ff:ff:ff:ff
4: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ens3 state UP mode DEFAULT group default qlen 1000
    link/ether 12:3f:bd:fd:bb:1e brd ff:ff:ff:ff:ff:ff

The culprit, as I ever indicated before, is that even with the seemingly correct netplan config provided, the netplan tool itself still doesn't recognize the 3-netdev failover interface, ending up with below udev rules renderred that runs conflict with the 3-netdev model:

# cat /run/systemd/network/10-netplan-ens3.link
[Match]
MACAddress=12:3f:bd:fd:bb:1e

[Link]
Name=ens3
WakeOnLan=off

# cat /run/udev/rules.d/99-netplan-ens3.rules
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="12:3f:bd:fd:bb:1e", NAME="ens3"

I understand that the solution we agreed upon has to rely on cloud-init providing supplemental udev rules for naming the ens3nsby (failover standby) and the ens4 (failover primary) interface. But, I don't see how the resulted 99-netplan-ens3.rules would work, given that all the 3 netdevs share a same MAC address, and there's no other udev attribute to differentiate? What's in your mind that needs to be fixed next, netplan or udevd?

Thanks,
-Siwei

[Jay Vosburgh (jvosburgh)]

Si-Wei,

What environment and methodology are you testing with? I do not see the same results you are reporting. I am using the instructions you previously provided, and with an 18.04.5 Ubuntu image, I see the expected network interface naming (ens3, ens3nsby), and do not see /run/systemd/network or /run/udev/rules.d as you describe.

[Si-Wei Liu (siwliu)]

Thanks, Jay. Sorry for getting back late.

I just tested it on an 18.04.5 Ubuntu image, and just as you mentioned it's working as expected. I think we can close this bug.

# ip l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens3: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 12:3f:bd:fd:bb:1e brd ff:ff:ff:ff:ff:ff
3: ens3nsby: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ens3 state UP mode DEFAULT group default qlen 1000
    link/ether 12:3f:bd:fd:bb:1e brd ff:ff:ff:ff:ff:ff
4: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ens3 state UP mode DEFAULT group default qlen 1000
    link/ether 12:3f:bd:fd:bb:1e brd ff:ff:ff:ff:ff:ff

[Mauricio Faria de Oliveira (mfo)]

Marking all other packages/dev release as Invalid.

Per previous discussion with Jay, iirc, those are
leftovers from a previously considered solution(s)
which was not the final one (in initramfs-tools.)

And just Bionic had to be patched as later releases
were not affected.

Please feel free to fix-up if that's not accurate.