ixgbe{vf} - Physical Function gets IRQ when VF checks link state

[Guilherme G. Piccoli]

[Impact]

* Intel NICs that are SR-IOV capable and are managed by ixgbe driver presents a potentially harmful behavior when the ixgbevf-managed VFs (Virtual Functions) perform an ethtool link check. The ixgbevf driver issues a mailbox command in the ethtool link state handler, which induces one IRQ in the PF (Physical Function) per link check.

* This was reported as a sort of "denial-of-service" from a guest; due to some link check loop running inside a guest with PCI-PT of a ixgbevf-managed VF, the host received a huge amount of IRQs causing soft-lockups.

* The patch proposed in this SRU request fix this behavior by relying in the saved link state (obtained in the ixgbevf's watchdog routine) instead of issuing a mailbox command to the PF in every link state check request. The commit is available on Linus tree: 1e1b0c658d9b ("ixgbevf: Use cached link state instead of re-reading the value for ethtool")
http://git.kernel.org/linus/1e1b0c658d9b

[Test case]

Reproducing the behavior is pretty simple; having a machine with an Intel NIC managed by ixgbe, proceed with the following steps:

a) Create one or more VFs
(echo 1 > /sys/class/net/<PF iface>/device/sriov_numvfs)

b) In a different terminal, monitor the non-TxRx PF IRQs:
(watch -n1 "cat /proc/interrupts | grep <PF iface> | grep -v Tx")

c) Run "ethtool <VF iface>" in a loop

Without the hereby proposed patch, the PF IRQs will increase.

[Regression potential]

The patch scope is restricted to ixgbevf ethtool link-check procedure, and was developed by the vendor itself. Being a self-contained patch affecting only this driver's ethtool handler, the worst potential regression would be a wrong link state report.

[Guilherme G. Piccoli (gpiccoli)]

SRU request was sent to kernel-team mailing list: https://lists.ubuntu.com/archives/kernel-team/2019-July/102282.html

Cheers,

Guilherme

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-disco' to 'verification-done-disco'. If the problem still exists, change the tag 'verification-needed-disco' to 'verification-failed-disco'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-xenial' to 'verification-done-xenial'. If the problem still exists, change the tag 'verification-needed-xenial' to 'verification-failed-xenial'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Guilherme G. Piccoli (gpiccoli)]

I've validated the -proposed kernels for Xenial (4.4.0-158), Bionic (4.15.0-56) and Disco (5.0.0-22), using the test case mentioned in the description. All working fine, the issue is gone.
Also, the patch was released upstream in the 5.3.x series, so I'll mark ff-series as Released.

Cheers,

Guilherme

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 5.2.0-10.11

---------------
linux (5.2.0-10.11) eoan; urgency=medium

  * eoan/linux: 5.2.0-10.11 -proposed tracker (LP: #1838113)

  * Packaging resync (LP: #1786013)
    - [Packaging] resync git-ubuntu-log

  * Eoan update: v5.2.4 upstream stable release (LP: #1838428)
    - bnx2x: Prevent load reordering in tx completion processing
    - caif-hsi: fix possible deadlock in cfhsi_exit_module()
    - hv_netvsc: Fix extra rcu_read_unlock in netvsc_recv_callback()
    - igmp: fix memory leak in igmpv3_del_delrec()
    - ipv4: don't set IPv6 only flags to IPv4 addresses
    - ipv6: rt6_check should return NULL if 'from' is NULL
    - ipv6: Unlink sibling route in case of failure
    - net: bcmgenet: use promisc for unsupported filters
    - net: dsa: mv88e6xxx: wait after reset deactivation
    - net: make skb_dst_force return true when dst is refcounted
    - net: neigh: fix multiple neigh timer scheduling
    - net: openvswitch: fix csum updates for MPLS actions
    - net: phy: sfp: hwmon: Fix scaling of RX power
    - net_sched: unset TCQ_F_CAN_BYPASS when adding filters
    - net: stmmac: Re-work the queue selection for TSO packets
    - net/tls: make sure offload also gets the keys wiped
    - nfc: fix potential illegal memory access
    - r8169: fix issue with confused RX unit after PHY power-down on RTL8411b
    - rxrpc: Fix send on a connected, but unbound socket
    - sctp: fix error handling on stream scheduler initialization
    - sctp: not bind the socket in sctp_connect
    - sky2: Disable MSI on ASUS P6T
    - tcp: be more careful in tcp_fragment()
    - tcp: fix tcp_set_congestion_control() use from bpf hook
    - tcp: Reset bytes_acked and bytes_received when disconnecting
    - vrf: make sure skb->data contains ip header to make routing
    - net/mlx5e: IPoIB, Add error path in mlx5_rdma_setup_rn
    - net: bridge: mcast: fix stale nsrcs pointer in igmp3/mld2 report handling
    - net: bridge: mcast: fix stale ipv6 hdr pointer when handling v6 query
    - net: bridge: don't cache ether dest pointer on input
    - net: bridge: stp: don't cache eth dest pointer before skb pull
    - macsec: fix use-after-free of skb during RX
    - macsec: fix checksumming after decryption
    - netrom: fix a memory leak in nr_rx_frame()
    - netrom: hold sock when setting skb->destructor
    - selftests: txring_overwrite: fix incorrect test of mmap() return value
    - net/tls: fix poll ignoring partially copied records
    - net/tls: reject offload of TLS 1.3
    - net/mlx5e: Fix port tunnel GRE entropy control
    - net/mlx5e: Rx, Fix checksum calculation for new hardware
    - net/mlx5e: Fix return value from timeout recover function
    - net/mlx5e: Fix error flow in tx reporter diagnose
    - bnxt_en: Fix VNIC accounting when enabling aRFS on 57500 chips.
    - mlxsw: spectrum_dcb: Configure DSCP map as the last rule is removed
    - net/mlx5: E-Switch, Fix default encap mode
    - mlxsw: spectrum: Do not process learned records with a dummy FID
    - dma-buf: balance refcount inbalance
    - dma-buf: Discard old fence_excl on retrying get_fences_rcu for realloc
    - Revert "gpio/spi: Fix spi-gpio...

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 5.0.0-25.26

---------------
linux (5.0.0-25.26) disco; urgency=medium

  * CVE-2019-1125
    - x86/cpufeatures: Carve out CQM features retrieval
    - x86/cpufeatures: Combine word 11 and 12 into a new scattered features word
    - x86/speculation: Prepare entry code for Spectre v1 swapgs mitigations
    - x86/speculation: Enable Spectre v1 swapgs mitigations
    - x86/entry/64: Use JMP instead of JMPQ
    - x86/speculation/swapgs: Exclude ATOMs from speculation through SWAPGS

 -- Kleber Sacilotto de Souza <email address hidden> Thu, 01 Aug 2019 12:04:35 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.15.0-58.64

---------------
linux (4.15.0-58.64) bionic; urgency=medium

  * unable to handle kernel NULL pointer dereference at 000000000000002c (IP:
    iget5_locked+0x9e/0x1f0) (LP: #1838982)
    - Revert "ovl: set I_CREATING on inode being created"
    - Revert "new primitive: discard_new_inode()"

linux (4.15.0-57.63) bionic; urgency=medium

  * CVE-2019-1125
    - x86/cpufeatures: Carve out CQM features retrieval
    - x86/cpufeatures: Combine word 11 and 12 into a new scattered features word
    - x86/speculation: Prepare entry code for Spectre v1 swapgs mitigations
    - x86/speculation: Enable Spectre v1 swapgs mitigations
    - x86/entry/64: Use JMP instead of JMPQ
    - x86/speculation/swapgs: Exclude ATOMs from speculation through SWAPGS

  * Packaging resync (LP: #1786013)
    - update dkms package versions

linux (4.15.0-56.62) bionic; urgency=medium

  * bionic/linux: 4.15.0-56.62 -proposed tracker (LP: #1837626)

  * Packaging resync (LP: #1786013)
    - [Packaging] resync git-ubuntu-log
    - [Packaging] update helper scripts

  * CVE-2019-2101
    - media: uvcvideo: Fix 'type' check leading to overflow

  * hibmc-drm Causes Unreadable Display for Huawei amd64 Servers (LP: #1762940)
    - [Config] Set CONFIG_DRM_HISI_HIBMC to arm64 only
    - SAUCE: Make CONFIG_DRM_HISI_HIBMC depend on ARM64

  * Bionic: support for Solarflare X2542 network adapter (sfc driver)
    (LP: #1836635)
    - sfc: make mem_bar a function rather than a constant
    - sfc: support VI strides other than 8k
    - sfc: add Medford2 (SFC9250) PCI Device IDs
    - sfc: improve PTP error reporting
    - sfc: update EF10 register definitions
    - sfc: populate the timer reload field
    - sfc: update MCDI protocol headers
    - sfc: support variable number of MAC stats
    - sfc: expose FEC stats on Medford2
    - sfc: expose CTPIO stats on NICs that support them
    - sfc: basic MCDI mapping of 25/50/100G link speeds
    - sfc: support the ethtool ksettings API properly so that 25/50/100G works
    - sfc: add bits for 25/50/100G supported/advertised speeds
    - sfc: remove tx and MCDI handling from NAPI budget consideration
    - sfc: handle TX timestamps in the normal data path
    - sfc: add function to determine which TX timestamping method to use
    - sfc: use main datapath for HW timestamps if available
    - sfc: only enable TX timestamping if the adapter is licensed for it
    - sfc: MAC TX timestamp handling on the 8000 series
    - sfc: on 8000 series use TX queues for TX timestamps
    - sfc: only advertise TX timestamping if we have the license for it
    - sfc: simplify RX datapath timestamping
    - sfc: support separate PTP and general timestamping
    - sfc: support second + quarter ns time format for receive datapath
    - sfc: support Medford2 frequency adjustment format
    - sfc: add suffix to large constant in ptp
    - sfc: mark some unexported symbols as static
    - sfc: update MCDI protocol headers
    - sfc: support FEC configuration through ethtool
    - sfc: remove ctpio_dmabuf_start from stats
    - sfc: stop the TX queue before pushing new buffers

  * [18.04 FEAT] zKVM: Add hardwar...

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.4.0-159.187

---------------
linux (4.4.0-159.187) xenial; urgency=medium

  * CVE-2019-1125
    - x86/cpufeatures: Carve out CQM features retrieval
    - x86/cpufeatures: Combine word 11 and 12 into a new scattered features word
    - x86/speculation: Prepare entry code for Spectre v1 swapgs mitigations
    - x86/speculation: Enable Spectre v1 swapgs mitigations
    - x86/entry/64: Use JMP instead of JMPQ
    - x86/speculation/swapgs: Exclude ATOMs from speculation through SWAPGS

linux (4.4.0-158.186) xenial; urgency=medium

  * xenial/linux: 4.4.0-158.186 -proposed tracker (LP: #1837609)

  * Packaging resync (LP: #1786013)
    - [Packaging] resync git-ubuntu-log
    - [Packaging] update helper scripts

  * ixgbe{vf} - Physical Function gets IRQ when VF checks link state
    (LP: #1836760)
    - ixgbevf: Use cached link state instead of re-reading the value for ethtool

  * CVE-2018-5383
    - crypto: kpp - Key-agreement Protocol Primitives API (KPP)
    - crypto: dh - Add DH software implementation
    - crypto: ecdh - Add ECDH software support
    - crypto: ecdh - make ecdh_shared_secret unique
    - crypto: doc - add KPP documentation
    - crypto: kpp, (ec)dh - fix typos
    - crypto: ecc - remove unused function arguments
    - crypto: ecc - remove unnecessary casts
    - crypto: ecc - rename ecdh_make_pub_key()
    - crypto: ecdh - add privkey generation support
    - crypto: ecc - Fix NULL pointer deref. on no default_rng
    - [Config] CRYPTO_ECDH=m
    - Bluetooth: convert smp and selftest to crypto kpp API
    - crypto: ecdh - add public key verification test

  * Xenial update: 4.4.185 upstream stable release (LP: #1836668)
    - fs/binfmt_flat.c: make load_flat_shared_library() work
    - scsi: vmw_pscsi: Fix use-after-free in pvscsi_queue_lck()
    - tracing: Silence GCC 9 array bounds warning
    - gcc-9: silence 'address-of-packed-member' warning
    - usb: chipidea: udc: workaround for endpoint conflict issue
    - Input: uinput - add compat ioctl number translation for UI_*_FF_UPLOAD
    - apparmor: enforce nullbyte at end of tag string
    - parport: Fix mem leak in parport_register_dev_model
    - parisc: Fix compiler warnings in float emulation code
    - IB/hfi1: Insure freeze_work work_struct is canceled on shutdown
    - MIPS: uprobes: remove set but not used variable 'epc'
    - net: hns: Fix loopback test failed at copper ports
    - sparc: perf: fix updated event period in response to PERF_EVENT_IOC_PERIOD
    - scripts/checkstack.pl: Fix arm64 wrong or unknown architecture
    - scsi: ufs: Check that space was properly alloced in copy_query_response
    - s390/qeth: fix VLAN attribute in bridge_hostnotify udev event
    - hwmon: (pmbus/core) Treat parameters as paged if on multiple pages
    - Btrfs: fix race between readahead and device replace/removal
    - btrfs: start readahead also in seed devices
    - can: flexcan: fix timeout when set small bitrate
    - can: purge socket error queue on sock destruct
    - ARM: imx: cpuidle-imx6sx: Restrict the SW2ISO increase to i.MX6SX
    - Bluetooth: Align minimum encryption key size for LE and BR/EDR connections
    - Bluet...

[Mauricio Faria de Oliveira (mfo)]

Verification successful on Ubuntu 14.04 (ESM):

No interrupts are generated on the PF by the link status check of the VF.
Verified with both Link down and Link up.

# uname -rv
3.13.0-174-generic #225-Ubuntu SMP Wed Oct 2 22:08:03 UTC 2019

# ls -ld /sys/bus/pci/drivers/ixgbe/*/net/*
drwxr-xr-x 5 root root 0 Oct 17 20:53 /sys/bus/pci/drivers/ixgbe/0000:04:00.0/net/eth4
drwxr-xr-x 5 root root 0 Oct 17 20:53 /sys/bus/pci/drivers/ixgbe/0000:04:00.1/net/eth5

# echo 1 > /sys/class/net/eth4/device/sriov_numvfs

# dmesg
...
[ 1104.000023] ixgbe 0000:04:00.0 eth4: SR-IOV enabled with 1 VFs
[ 1104.100390] pci 0000:04:10.0: [8086:1515] type 00 class 0x020000
[ 1104.100981] ixgbe 0000:04:00.0: removed PHC on eth4
[ 1104.109683] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 2.11.3-k
[ 1104.109689] ixgbevf: Copyright (c) 2009 - 2012 Intel Corporation.
[ 1104.109797] ixgbevf 0000:04:10.0: enabling device (0000 -> 0002)
[ 1104.109976] ixgbe 0000:04:00.0 eth4: VF Reset msg received from vf 0
[ 1104.110096] ixgbe 0000:04:00.0: VF 0 has no MAC address assigned, you may have to assign one manually
[ 1104.120353] ixgbevf 0000:04:10.0: PF still in reset state. Is the PF interface up?
[ 1104.120359] ixgbevf 0000:04:10.0: Assigning random MAC address
[ 1104.121016] ixgbevf 0000:04:10.0: irq 305 for MSI/MSI-X
[ 1104.121037] ixgbevf 0000:04:10.0: irq 306 for MSI/MSI-X
[ 1104.516319] ixgbe 0000:04:00.0: irq 155 for MSI/MSI-X
[ 1104.516335] ixgbe 0000:04:00.0: irq 156 for MSI/MSI-X
[ 1104.516359] ixgbe 0000:04:00.0: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1
[ 1104.523447] pps pps1: new PPS source ptp4
[ 1104.523452] ixgbe 0000:04:00.0: registered PHC device on eth4
[ 1111.947668] ixgbe 0000:04:00.0 eth4: NIC Link is Up 10 Gbps, Flow Control: None

# ls -ld /sys/bus/pci/drivers/ixgbevf/*/net/*
drwxr-xr-x 5 root root 0 Oct 17 21:11 /sys/bus/pci/drivers/ixgbevf/0000:04:10.0/net/enp4s16

Link down:

# ethtool enp4s16 | grep Link
 Link detected: no

# while true; do ethtool enp4s16 >/dev/null; done

On another terminal:

$ while sleep 1; do echo -n "$(date) - "; cat /proc/interrupts | grep eth4 | grep -v Tx | tr -s ' '; done
Thu Oct 17 21:21:08 UTC 2019 - 156: 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth4
Thu Oct 17 21:21:09 UTC 2019 - 156: 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth4
Thu Oct 17 21:21:10 UTC 2019 - 156: 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth4
Thu Oct 17 21:21:11 UTC 2019 - 156: 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth4
Thu Oct 17 21:21:12 UTC 2019 - 156: 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth4
Thu Oct 17 21:21:13 UTC 2019 - 156: 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-e...