hot add VF to net_failover - could not rename interface '8' from 'eth0' to 'ens4': Device or resource busy

[Si-Wei Liu]

[Impact]

udev fails to rename a new interface when a VF is added due to a race between the kernel and userspace.

It's desirable to get a consistent and predictable name, as otherwise any persistent configuration in userspace can't be applied properly on the Virtual Function that gets hot plugged in.

[Test Case]

Host has a QEMU/KVM setup with standby virtio-net [1]:

# qemu-system-x86_64 -name guest=ubuntu-xenial,debug-threads=on -machine pc,accel=kvm,usb=off -cpu host -m 4096 -realtime mlock=off -smp 4,sockets=1,cores=2,threads=2 -monitor stdio -vnc :1 -boot order=n,menu=on -serial telnet:127.0.0.1:4567,server -netdev tap,fd=20,id=hostnet0,vhost=on,vhostfd=21 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=d6:36:08:7f:b4:d9,standby=on

Guest is loaded with Xenial Xerus (16.04.5),

vsbalakr@ubuntu-16:~$ uname -a
Linux ubuntu-16 4.15.0-1007-oracle #9~16.04.1-Ubuntu SMP Wed Dec 12 19:49:55 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
vsbalakr@ubuntu-16:~$ cat /proc/version_signature
Ubuntu 4.15.0-1007.9~16.04.1-oracle 4.15.18
vsbalakr@ubuntu-16:~$ cat /etc/os*release
NAME="Ubuntu"
VERSION="16.04.5 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.5 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
vsbalakr@ubuntu-16:~$

ens3 is the master interface of net_failover, while ens3nsby is its standby slave [2]:

vsbalakr@ubuntu-16:~$ ip a sh
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether d6:36:08:7f:b4:d9 brd ff:ff:ff:ff:ff:ff
    inet 10.211.15.21/21 brd 10.211.15.255 scope global ens3
       valid_lft forever preferred_lft forever
    inet6 fe80::d436:8ff:fe7f:b4d9/64 scope link
        valid_lft forever preferred_lft forever
3: ens3nsby: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ens3 state UP group default qlen 1000
    link/ether d6:36:08:7f:b4:d9 brd ff:ff:ff:ff:ff:ff
    inet 10.211.15.21/21 brd 10.211.15.255 scope global dynamic ens3nsby
       valid_lft 2154sec preferred_lft 2154sec
    inet6 fe80::d436:8ff:fe7f:b4d9/64 scope link
        valid_lft forever preferred_lft forever

Now we hot plug a Virtual Function (with MAC set to same address d6:36:08:7f:b4:d9 in prior) into the guest, via QEMU HMP console:

(qemu) device_add vfio-pci,host=02:10.1,id=hostdev_vf0
(qemu)

VF now shows up in guest as "eth0" instead of the expected "ens4":

vsbalakr@ubuntu-16:~$ ip a sh
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether d6:36:08:7f:b4:d9 brd ff:ff:ff:ff:ff:ff
    inet 10.211.15.21/21 brd 10.211.15.255 scope global ens3
       valid_lft forever preferred_lft forever
    inet6 2606:b400:400:3451:61b8:3bbd:dba3:7f93/64 scope global temporary dynamic valid_lft 604753sec preferred_lft 86209sec
    inet6 2606:b400:400:3451:d436:8ff:fe7f:b4d9/64 scope global mngtmpaddr dynamic valid_lft 2591959sec preferred_lft 604759sec
    inet6 fe80::d436:8ff:fe7f:b4d9/64 scope link
       valid_lft forever preferred_lft forever
3: ens3nsby: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ens3 state UP group default qlen 1000
    link/ether d6:36:08:7f:b4:d9 brd ff:ff:ff:ff:ff:ff
    inet 10.211.15.21/21 brd 10.211.15.255 scope global dynamic ens3nsby
       valid_lft 2072sec preferred_lft 2072sec
    inet6 fe80::d436:8ff:fe7f:b4d9/64 scope link
       valid_lft forever preferred_lft forever
6: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ens3 state UP group default qlen 1000
    link/ether d6:36:08:7f:b4:d9 brd ff:ff:ff:ff:ff:ff
vsbalakr@ubuntu-16:~$

/var/log/syslog shows that renaming to "ens4" had failed because of "Device or resource busy",

Feb 8 18:24:05 ubuntu-16 kernel: [ 5270.231623] ixgbevf 0000:00:04.0: NIC Link is Up 10 Gbps
Feb 8 18:24:05 ubuntu-16 kernel: [ 5270.233188] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Feb 8 18:24:05 ubuntu-16 systemd-udevd[2340]: RUN 'ifupdown-hotplug' /lib/udev/rules.d/80-ifupdown.rules:5
Feb 8 18:24:05 ubuntu-16 systemd-udevd[2340]: IMPORT builtin 'path_id' /lib/udev/rules.d/80-net-setup-link.rules:5
Feb 8 18:24:05 ubuntu-16 systemd-udevd[2340]: IMPORT builtin 'net_setup_link' /lib/udev/rules.d/80-net-setup-link.rules:9
Feb 8 18:24:05 ubuntu-16 systemd-udevd[2340]: Config file /lib/systemd/network/99-default.link applies to device eth0
Feb 8 18:24:05 ubuntu-16 systemd-udevd[2340]: NAME 'ens4' /lib/udev/rules.d/80-net-setup-link.rules:11
Feb 8 18:24:05 ubuntu-16 systemd-udevd[2340]: RUN '/lib/systemd/systemd-sysctl --prefix=/net/ipv4/conf/$name --prefix=/net/ipv4/neigh/$name --prefix=/net/ipv6/conf/$name --prefix=/net/ipv6/neigh/$name' /lib/udev/rules.d/99-systemd.rules:51
Feb 8 18:24:05 ubuntu-16 systemd-udevd[2340]: error changing net interface name 'eth0' to 'ens4': Device or resource busy
Feb 8 18:24:05 ubuntu-16 kernel: [ 5270.236275] virtio_net virtio0 ens3: failover primary slave:eth0 registered
Feb 8 18:24:05 ubuntu-16 kernel: [ 5270.236308] ixgbevf 0000:00:04.0: d6:36:08:7f:b4:d9
Feb 8 18:24:05 ubuntu-16 kernel: [ 5270.236310] ixgbevf 0000:00:04.0: MAC: 4
Feb 8 18:24:05 ubuntu-16 kernel: [ 5270.236313] ixgbevf 0000:00:04.0: Intel(R) 82599 Virtual Function
Feb 8 18:24:05 ubuntu-16 systemd-udevd[2340]: could not rename interface '8' from 'eth0' to 'ens4': Device or resource busy
Feb 8 18:24:05 ubuntu-16 systemd-udevd[2340]: created db file '/run/udev/data/n8' for '/devices/pci0000:00/0000:00:04.0/net/eth0'
Feb 8 18:24:05 ubuntu-16 NetworkManager[1348]: <warn> [1549679045.5610] device (eth0): failed to find device 8 'eth0' with udev
Feb 8 18:24:05 ubuntu-16 systemd-udevd[2341]: starting 'vlan-network-interface UDEV'
Feb 8 18:24:05 ubuntu-16 NetworkManager[1348]: <info> [1549679045.5620] manager: (eth0): new Ethernet device (/org/freedesktop/NetworkManager/Devices/7)
Feb 8 18:24:05 ubuntu-16 NetworkManager[1348]: <info> [1549679045.5621] device (eth0): enslaved to non-master-type device ens3; ignoring
Feb 8 18:24:05 ubuntu-16 systemd-udevd[516]: could not create device: Invalid argument

it's desirable to get a consistent and preditable name, as otherwise any persistent configuration in userspace can't be applied properly on the Virtual Function that gets hot plugged in.

[1] https://patchwork.kernel.org/patch/10384967/
[2] https://www.kernel.org/doc/html/latest/networking/net_failover.html
---
ApportVersion: 2.20.1-0ubuntu2.18
Architecture: amd64
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
DistroRelease: Ubuntu 16.04
EcryptfsInUse: Yes
HibernationDevice: RESUME=UUID=486b7d46-3258-4d24-b387-948ce8d0bc9f
Lsusb: Error: command ['lsusb'] failed with exit code 1:
MachineType: QEMU Standard PC (i440FX + PIIX, 1996)
Package: linux (not installed)
ProcEnviron:
 TERM=dtterm
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-1007-oracle root=UUID=12bfaded-59ea-4a0a-b3ac-8c7a057ed82a ro ip=dhcp debug splash
ProcVersionSignature: Ubuntu 4.15.0-1007.9~16.04.1-oracle 4.15.18
RelatedPackageVersions:
 linux-restricted-modules-4.15.0-1007-oracle N/A
 linux-backports-modules-4.15.0-1007-oracle N/A
 linux-firmware 1.157.21
RfKill:

Tags: xenial xenial
Uname: Linux 4.15.0-1007-oracle x86_64
UnreportableReason: The report belongs to a package that is not installed.
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm cdrom dip lpadmin lxd plugdev sambashare sudo
_MarkForUpload: False
dmi.bios.date: 04/01/2014
dmi.bios.vendor: SeaBIOS
dmi.bios.version: 1.11.0-2.el7
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-i440fx-3.0
dmi.modalias: dmi:bvnSeaBIOS:bvr1.11.0-2.el7:bd04/01/2014:svnQEMU:pnStandardPC(i440FX+PIIX,1996):pvrpc-i440fx-3.0:cvnQEMU:ct1:cvrpc-i440fx-3.0:
dmi.product.name: Standard PC (i440FX + PIIX, 1996)
dmi.product.version: pc-i440fx-3.0
dmi.sys.vendor: QEMU

[Regression Potential]

The proposed solution introduces low risk of regression because it only affects the linux-oracle kernel on instances using net_failover. In case of regressions it's also possible to disable the new behaviour via the kernel cmdline.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1815268

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Si-Wei Liu (siwliu)]

apport information

[Si-Wei Liu (siwliu)]

apport information

[Si-Wei Liu (siwliu)]

apport information

[Si-Wei Liu (siwliu)]

apport information

[Si-Wei Liu (siwliu)]

apport information

[Si-Wei Liu (siwliu)]

apport information

[Si-Wei Liu (siwliu)]

apport information

[Si-Wei Liu (siwliu)]

apport information

[Si-Wei Liu (siwliu)]

apport information

[Si-Wei Liu (siwliu)]

apport information

[Si-Wei Liu (siwliu)]

apport information

[Si-Wei Liu (siwliu)]

apport information

[Si-Wei Liu (siwliu)]

Please find attached a patch (from Oracle Linux) that mitigates the race with a tunable delay of 100ms.

[Si-Wei Liu (siwliu)]

Note the patch attached is a tentative workaround that helps mitigate the issue, but not intended to be a suggested fix. I've initiated a thread in the Linux netdev mailing list for the resolution. Please find the relevant thread starting at:

https://<email address hidden>/msg286707.html

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-oracle - 4.15.0-1010.12

---------------
linux-oracle (4.15.0-1010.12) bionic; urgency=medium

  * linux-oracle: 4.15.0-1010.12 -proposed tracker (LP: #1819704)

  * hot add VF to net_failover - could not rename interface '8' from 'eth0' to
    'ens4': Device or resource busy (LP: #1815268)
    - SAUCE: net_failover: delay taking over primary device to accommodate udevd
      renaming

  [ Ubuntu: 4.15.0-47.50 ]

  * linux: 4.15.0-47.50 -proposed tracker (LP: #1819716)
  * Packaging resync (LP: #1786013)
    - [Packaging] resync getabis
    - [Packaging] update helper scripts
    - [Packaging] resync retpoline extraction
  * C++ demangling support missing from perf (LP: #1396654)
    - [Packaging] fix a mistype
  * arm-smmu-v3 arm-smmu-v3.3.auto: CMD_SYNC timeout (LP: #1818162)
    - iommu/arm-smmu-v3: Fix unexpected CMD_SYNC timeout
  * Crash in nvme_irq_check() when using threaded interrupts (LP: #1818747)
    - nvme-pci: fix out of bounds access in nvme_cqe_pending
  * CVE-2019-9213
    - mm: enforce min addr even if capable() in expand_downwards()
  * CVE-2019-3460
    - Bluetooth: Check L2CAP option sizes returned from l2cap_get_conf_opt
  * amdgpu with mst WARNING on blanking (LP: #1814308)
    - drm/amd/display: Don't use dc_link in link_encoder
    - drm/amd/display: Move wait for hpd ready out from edp power control.
    - drm/amd/display: eDP sequence BL off first then DP blank.
    - drm/amd/display: Fix unused variable compilation error
    - drm/amd/display: Fix warning about misaligned code
    - drm/amd/display: Fix MST dp_blank REG_WAIT timeout
  * tun/tap: unable to manage carrier state from userland (LP: #1806392)
    - tun: implement carrier change
  * CVE-2019-8980
    - exec: Fix mem leak in kernel_read_file
  * raw_skew in timer from the ubuntu_kernel_selftests failed on Bionic
    (LP: #1811194)
    - selftest: timers: Tweak raw_skew to SKIP when ADJ_OFFSET/other clock
      adjustments are in progress
  * [Packaging] Allow overlay of config annotations (LP: #1752072)
    - [Packaging] config-check: Add an include directive
  * CVE-2019-7308
    - bpf: move {prev_,}insn_idx into verifier env
    - bpf: move tmp variable into ax register in interpreter
    - bpf: enable access to ax register also from verifier rewrite
    - bpf: restrict map value pointer arithmetic for unprivileged
    - bpf: restrict stack pointer arithmetic for unprivileged
    - bpf: restrict unknown scalars of mixed signed bounds for unprivileged
    - bpf: fix check_map_access smin_value test when pointer contains offset
    - bpf: prevent out of bounds speculation on pointer arithmetic
    - bpf: fix sanitation of alu op with pointer / scalar type from different
      paths
    - bpf: add various test cases to selftests
  * CVE-2017-5753
    - bpf: properly enforce index mask to prevent out-of-bounds speculation
    - bpf: fix inner map masking to prevent oob under speculation
  * BPF: kernel pointer leak to unprivileged userspace (LP: #1815259)
    - bpf/verifier: disallow pointer subtraction
  * squashfs hardening (LP: #1816756)
    - squashfs: more metadata hardening
    - squashfs metadata 2: electric bo...